Databricks-DE-Pro Practice Exam 3 — 20 Questions

Question 1

A data engineer is configuring a Databricks Workflow that must run a notebook task only after a previous task that writes to a Delta table has completed successfully. The engineer wants to ensure that if the first task fails, the second task does not run. Which feature should the engineer use to define this dependency?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the depends_on property of the second task to reference the first task's task_key.. Task dependencies in Databricks Workflows are defined using the depends_on property, which lists upstream task keys. This ensures the dependent task runs only after the specified upstream tasks complete successfully. It is the fundamental way to build a directed acyclic graph (DAG) of tasks within a job.

Answer

Configure the second task with a run_if condition set to ALL_SUCCESS.

Answer

Use a trigger to start the second task when the first task completes, using a file arrival trigger on the Delta table location.

Answer

Add a condition task between the two tasks that checks the Delta table for new records and only proceeds if records exist.

Question 2

An engineer is writing a Python function to process data in a Databricks Notebook. Which command should they use to ensure that secrets, such as API keys, are not hardcoded or exposed in the plain text of the notebook?

Accepted Answer

dbutils.secrets.get(scope='my_scope', key='my_key'). The dbutils.secrets.get() utility is the standard, secure way to retrieve sensitive information stored in Databricks Secret Scopes. By referencing the scope and key name, the secret value is fetched at runtime and remains masked in the notebook output, preventing accidental exposure of credentials. This is a mandatory practice in any secure data engineering environment to comply with security policies and prevent unauthorized access to downstream data sources.

Answer

os.environ.get('API_KEY')

Answer

open('/secret/path').read()

Answer

spark.conf.get('secret_key')

Question 3

Refer to the exhibit. A data engineer creates an instance pool to reduce cluster startup times for development teams. However, finance reports indicate unexpected cloud infrastructure charges. Based on the configuration shown in the exhibit, what is the primary driver of these unexpected costs?

Accepted Answer

The min_idle_instances setting maintains running virtual machines continuously, incurring persistent infrastructure costs.. Instance pools maintain idle virtual machines ready for immediate attachment to clusters, drastically reducing startup latency. However, setting min_idle_instances to 2 ensures that two instances are running constantly even when no clusters are active, incurring continuous cloud provider infrastructure charges and idle DBU fees that accumulate rapidly.

Answer

The max_capacity limit is set too low, forcing teams to provision multiple competing instance pools.

Answer

Instance pools do not support Spot instances, forcing all pool-backed clusters to run expensive on-demand VMs.

Answer

The selected node_type_id is optimized for storage rather than compute, causing inflated licensing surcharges.

Question 4

Refer to the exhibit. Which action is the most appropriate to resolve this memory-related failure during the job execution?

Accepted Answer

Upgrade the cluster to an instance type with higher memory capacity.. The error indicates an Out of Memory (OOM) condition where the container exceeded its allocated memory limit. Increasing the instance type to a memory-optimized VM size provides more RAM per executor, allowing Spark to process larger partitions without triggering the YARN memory killer. This is a common bottleneck in memory-intensive operations like wide transformations, where adjusting the cluster configuration is more effective than attempting to optimize complex query logic.

Answer

Decrease the number of partitions using spark.sql.shuffle.partitions.

Answer

Enable autoscaling to add more nodes to the cluster.

Answer

Change the file format from Parquet to CSV for better performance.

Question 5

Refer to the exhibit. The alert is intended to trigger if the data in 'my_table' is older than one hour. Which query modification correctly implements this check?

Accepted Answer

SELECT CASE WHEN max(updated_at) < now() - interval 1 hour THEN 1 ELSE 0 END. To detect stale data, the alert must compare the latest record timestamp against the current time. If the latest 'updated_at' value is less than the threshold (one hour ago), it signifies that no data has arrived in the last hour. Monitoring data freshness is critical for time-sensitive analytics, ensuring that business stakeholders are working with current data and allowing teams to troubleshoot ingestion pipeline failures proactively.

Answer

SELECT COUNT(*) FROM my_table WHERE updated_at < now() - interval 1 hour

Answer

SELECT updated_at FROM my_table ORDER BY updated_at DESC LIMIT 1

Answer

SELECT datediff(now(), max(updated_at)) FROM my_table

Question 6

Refer to the exhibit. You are appending data to an existing Delta table. What is the most likely cause of this error, and how should you resolve it?

Accepted Answer

The incoming data's 'price' column has a higher precision than the table schema.. This error occurs because of a data type mismatch between the incoming DataFrame and the existing table schema, specifically a change in precision or scale. Databricks enforces schema safety to prevent data corruption. Resolving this requires either casting the incoming data to match the target schema or using the 'mergeSchema' option if the goal is to allow evolution, provided the change is safe and intended for the application.

Answer

The table is locked by another process.

Answer

The table schema is corrupted and needs to be repaired.

Answer

The user does not have write access to the table.

Question 7

A Data Engineer is building a Lakeflow Spark Declarative Pipelines pipeline that ingests JSON sensor events. The pipeline must drop records where the `sensor_id` is NULL, ensure that `event_time` is not in the future, and continue processing without failing the update. Which combination of expectations should be used?

Accepted Answer

Use `@dp.expect_or_drop("valid_sensor", "sensor_id IS NOT NULL")` and `@dp.expect_or_drop("valid_time", "event_time <= current_timestamp()")` on the dataset.. The pipeline must drop records that violate the conditions and continue processing. `expect_or_drop` is designed for this: it discards invalid rows and allows the update to succeed. Using `expect_or_fail` would halt the pipeline, and `expect_all` would not remove the bad records, leaving them in the target table.

Answer

Use `@dp.expect_all({"valid_sensor": "sensor_id IS NOT NULL", "valid_time": "event_time <= current_timestamp()"})` on the dataset.

Answer

Use `@dp.expect_or_fail("valid_sensor", "sensor_id IS NOT NULL")` and `@dp.expect_or_drop("valid_time", "event_time <= current_timestamp()")` on the dataset.

Answer

Use `@dp.expect_all_or_fail({"valid_sensor": "sensor_id IS NOT NULL", "valid_time": "event_time <= current_timestamp()"})` on the dataset.

Question 8

Refer to the exhibit. An engineer created this alert for a query. Under what condition will the alert status change to 'Triggered'?

Accepted Answer

When the 'duration' column value in the query output is strictly greater than 3600.. The alert is configured to monitor the 'duration' column from the specified query result. If the returned value exceeds 3600, the alert enters the 'Triggered' state. Understanding this threshold-based mechanism is vital for maintaining pipeline performance. If queries consistently exceed this time, the alert notifies the engineer to investigate potential resource contention, suboptimal query plans, or data volume spikes, ensuring the stability and timely execution of data processing tasks.

Answer

When the query returns a row count greater than 3600.

Answer

When the query execution takes longer than 3600 seconds to run.

Answer

When the query fails and returns an error code equal to 3600.

Question 9

A data engineer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must handle late-arriving data and update already processed aggregates. Which watermark strategy should be used to allow updates to aggregates while bounding state store growth?

Accepted Answer

Use a watermark on the event-time column with a delay that accommodates the maximum expected lateness, and set the output mode to 'update'.. The correct answer is to use an event-time watermark with a delay that matches the maximum expected lateness and to use the 'update' output mode. This allows late data to be incorporated into aggregates while bounding state store growth. The watermark tells the engine how long to wait for late data before finalizing a window, and 'update' mode emits incremental updates, which is ideal for updating aggregates. Other options either use inappropriate time semantics or set delays that are too short.

Answer

Use a watermark based on processing time rather than event time, with a 10-minute delay.

Answer

Use a watermark on the event-time column with a delay of 0 seconds to ensure all data is processed immediately.

Answer

Use a static watermark of 1 hour by setting the watermark delay to '1 hour' on the event-time column.

Question 10

A data engineer is preparing to deploy a production Databricks workflow. Which TWO best practices should be implemented to ensure maintainability and robust error handling?

Accepted Answer

Use Databricks Git folders for version control of production code.. Maintaining production environments requires strict adherence to modular code design and comprehensive observability. Using version control for notebooks and job configurations ensures that every change is tracked, audited, and reversible. Simultaneously, implementing robust notification alerts for job failures allows engineering teams to respond proactively to issues. These two practices collectively minimize the risk of deployment errors and reduce the Mean Time to Resolution (MTTR) when unexpected failures occur in production pipelines.

Answer

Hardcode credentials directly into the notebook for ease of access.

Answer

Avoid using libraries and rely only on built-in Spark functions.

Answer

Deploy code directly from the workspace to production.

Question 11

Which THREE features are provided by Delta Lake when compared to standard Parquet files? (Select THREE)

Accepted Answer

ACID transactions. Delta Lake adds a transaction log (the _delta_log) to standard Parquet files, enabling ACID transactions, time travel, and schema enforcement. These features solve common data engineering pain points such as partial writes, data corruption, and the inability to revert to previous versions of data. Mastering these capabilities is essential for building reliable data lakes that provide the same consistency and reliability as traditional data warehouses.

Answer

Automatic indexing for all columns

Answer

Built-in support for real-time web socket streaming

Question 12

What is the primary function of a 'Personal Access Token' (PAT) in Databricks, and why is it considered a security risk if not managed properly?

Accepted Answer

It provides long-lived programmatic access to the API.. Personal Access Tokens (PATs) are used for programmatic access to Databricks REST APIs. They authenticate the user and inherit their permissions. The risk lies in their longevity; if an unexpired token is leaked, an attacker can impersonate the user without needing to re-authenticate via SSO. Therefore, limiting token duration and using service principals for automated tasks are crucial security measures to protect the platform.

Answer

It allows users to bypass multi-factor authentication.

Answer

It is intended for end-user dashboard access.

Answer

It encrypts data stored in the workspace.

Question 13

Refer to the exhibit. A Databricks administrator wants to restrict access to a specific SQL Alert. Based on the JSON policy, which statement accurately describes the current permission model for this alert?

Accepted Answer

The user can see the alert, but cannot trigger a manual refresh or modify the alert settings.. The exhibit displays an Access Control List (ACL) entry granting 'CAN_VIEW' permission to a specific user for a single alert resource. In Databricks, SQL Alerts are secured via the workspace-level permission model. Understanding these JSON-based policy definitions is essential for data engineers tasked with auditing governance and ensuring that sensitive monitoring data is only accessible to authorized personnel during security reviews.

Answer

The user is allowed to modify the alert's query criteria and notification frequency.

Answer

The user has sufficient permissions to delete the alert from the Databricks SQL workspace.

Answer

This JSON policy provides the user with global access to all alerts within the SQL warehouse.

Question 14

Your organization is ingesting sensitive PII data. You need to ensure that personal identifiers are masked during the ingestion process before they are stored in the Bronze layer of your Medallion architecture. What is the best practice for this?

Accepted Answer

Apply masking logic within the initial streaming ingestion transformation.. Implementing masking at the ingestion layer using Delta Live Tables (DLT) expectations or standard Spark transformations ensures that sensitive data is never written in plaintext to the Bronze layer. Applying transformations during the 'Acquisition' phase is a critical security practice, ensuring that governance requirements are met before data becomes available to analysts, thereby reducing the risk of accidental exposure and maintaining data privacy compliance across the pipeline.

Answer

Mask the data using Delta Lake column masking after the data reaches the Silver layer.

Answer

Configure Unity Catalog to mask columns only for specific users.

Answer

Use a post-ingestion job to delete PII columns.

Question 15

What is the primary benefit of the Medallion architecture in a Databricks Lakehouse?

Accepted Answer

It provides a clear progression of data quality and structure.. The Medallion architecture provides a clear, structured progression of data quality. By segregating data into Bronze (raw), Silver (cleaned), and Gold (refined) layers, organizations can maintain an immutable audit trail while enabling both technical teams and business analysts to consume the data at the appropriate level of abstraction. This structure simplifies data governance, incremental processing, and data quality management, which are fundamental to building a reliable Lakehouse.

Answer

It eliminates the need for data partitioning and Z-Ordering.

Answer

It forces all data to be stored in a star schema at all layers.

Answer

It automatically converts all incoming data to a structured format.

Question 16

Which metric should a data engineer prioritize when investigating a slow-running query in the Databricks SQL query history?

Accepted Answer

The total execution time and breakdown of the query stages.. The 'Total Time' metric is decomposed into wait time, compilation time, and execution time. Identifying the execution time bottleneck helps determine if the issue is compute-bound, I/O-bound, or due to slow data retrieval. This information is vital for deciding whether to optimize the query code, adjust partitioning, or scale the warehouse, ensuring that engineering efforts are targeted at the right performance issues to maximize impact.

Answer

The number of users logged into the workspace at the time.

Answer

The color of the query status light in the UI.

Answer

The name of the user who submitted the query.

Question 17

Refer to the exhibit. A Data Engineer is attempting to merge data into a table with these constraints defined. If the incoming batch contains rows that violate these rules, what is the default behavior of the Delta Lake engine during the merge operation?

Accepted Answer

The transaction fails entirely, and no changes are committed to the Delta table.. Delta Lake maintains strict ACID compliance. When a transaction violates defined CHECK or NOT NULL constraints, the engine throws an error, and the entire transaction is rolled back. This behavior prevents data corruption and ensures that only valid state transitions occur. This is essential for maintaining data integrity in complex ETL pipelines where maintaining a 'known good' state is preferred over partial updates or silent failures.

Answer

The engine silently ignores the invalid rows and completes the merge for valid records only.

Answer

The invalid rows are automatically routed to a hidden sidecar table for later review.

Answer

The engine automatically updates the invalid rows to the default column value.

Question 18

A Data Engineer is implementing a medallion architecture. Which THREE steps are critical for effectively implementing a high-quality 'Silver' layer from 'Bronze' data?

Accepted Answer

Enforce strict schema validation on incoming Bronze files.. The Silver layer is the 'single source of truth' for refined data. By applying strict schema enforcement, deduplication, and standardized data types, you ensure that analytical tools receive reliable, high-quality information. Proper implementation here prevents the 'garbage in, garbage out' scenario, allowing data scientists and analysts to focus on modeling and reporting rather than constant, redundant data cleaning tasks, significantly increasing organizational productivity and trust in data.

Answer

Apply business logic and complex transformations to create aggregate summary tables.

Answer

Convert all column names to uppercase to ensure case-insensitive consistency.

Question 19

You need to perform a deduplication task on a streaming source that includes late-arriving data. Which Delta Lake feature is best suited to manage this while ensuring efficient state cleanup?

Accepted Answer

Use the dropDuplicates() method with watermark settings to manage state.. The `withWatermark` and `dropDuplicates` combination is specifically designed for streaming deduplication. Watermarks tell the engine how long to wait for late-arriving data, allowing it to clear old state from memory. This is critical for high-volume streaming jobs where state growth would otherwise lead to out-of-memory errors or significant performance degradation, ensuring the job remains stable over long periods of execution.

Answer

Use a standard SQL DELETE query with a subquery to identify and remove duplicates.

Answer

Set the table property 'delta.enableChangeDataFeed' to true and filter on the change log.

Answer

Increase the 'spark.sql.shuffle.partitions' setting to ensure all duplicates land on the same node.

Question 20

Which of the following describes the correct behavior of Unity Catalog's 'Data Lineage' when used for security compliance?

Accepted Answer

It captures table-to-table data dependencies.. Data Lineage in Unity Catalog automatically captures the flow of data from source to target. This is invaluable for security compliance, as it allows administrators to trace sensitive data across the environment, identify which tables are derived from PII sources, and understand the impact of modifying or deleting specific tables. By visualizing these dependencies, teams can ensure that security policies are consistently applied throughout the data lifecycle.

Answer

It shows which users accessed the data.

Answer

It manually triggers an alert when PII is detected.

Answer

It permanently stores the raw data content.