Databricks-DE-Pro Practice Test 13 — 15 Questions

Question 1

A data engineer needs to audit which users have accessed a Unity Catalog table containing sensitive data. They want to see a record of all queries that read from the table over the past 30 days. Which Unity Catalog feature should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Audit logs. Audit logs in Unity Catalog record all access and query activities, including the user, timestamp, and the tables accessed. They are essential for security auditing and compliance. Data lineage, information schema, and Delta transaction logs serve different purposes and do not provide a record of user access.

Answer

Delta transaction log

Answer

Data lineage

Answer

Information schema

Question 2

A data engineer is developing a PySpark job that reads from a Delta table and performs a series of transformations. The engineer notices that the job is slow and suspects that the query plan is not optimized because statistics are outdated. Which command should the engineer run to update the statistics for the Delta table to improve query performance?

Accepted Answer

ANALYZE TABLE table_name COMPUTE STATISTICS. The correct command is ANALYZE TABLE table_name COMPUTE STATISTICS. This command gathers statistics such as row count, column cardinality, and min/max values, which the Spark optimizer uses to generate efficient query plans. Outdated statistics can cause poor join orders or unnecessary shuffles. OPTIMIZE, VACUUM, and REFRESH TABLE serve different purposes and do not update statistics. Therefore, ANALYZE TABLE is the appropriate action to improve performance when statistics are stale.

Answer

VACUUM table_name

Answer

REFRESH TABLE table_name

Answer

OPTIMIZE table_name

Question 3

A data engineer is using PySpark to cleanse a large dataset of customer records. The DataFrame `df` contains a string column `phone` with values like '123-456-7890', '(123) 456-7890', and '1234567890'. The engineer needs to standardize these to digits only (e.g., '1234567890'). Which transformation should be used?

Accepted Answer

df.withColumn('phone_clean', regexp_replace('phone', '[^0-9]', '')). To standardize phone numbers to digits only, all non-digit characters must be removed. The regexp_replace function with the pattern '[^0-9]' efficiently replaces any character that is not a digit with an empty string, resulting in a clean digit-only string. This method is robust and handles various formats without complex string manipulation. Other functions like translate or split do not directly produce the desired output.

Answer

df.withColumn('phone_clean', split('phone', '[^0-9]'))

Answer

df.withColumn('phone_clean', translate('phone', '-() ', ''))

Answer

df.withColumn('phone_clean', trim('phone'))

Question 4

A data engineer is configuring audit logging for a Unity Catalog-enabled workspace. The security team wants to capture all access to tables and the granting of privileges. Which Databricks feature should the engineer enable to collect these audit events?

Accepted Answer

Diagnostic log delivery to a cloud storage location.. Databricks audit logs, delivered through diagnostic log delivery, capture Unity Catalog data access and privilege changes. They are the authoritative source for security auditing and compliance. Other features like Delta Live Tables event logs, data lineage, and workspace ACLs serve different purposes and do not provide the required audit event stream.

Answer

Workspace access control lists (ACLs).

Answer

Delta Live Tables event log.

Answer

Unity Catalog data lineage.

Question 5

A data engineer supports a Delta Live Tables pipeline that ingests streaming data from Kafka. The pipeline sometimes experiences latency spikes, and the engineer needs to determine whether the bottleneck is in the ingestion stage or in downstream transformations. They want to use built-in observability without adding external tooling. Which approach provides the most direct insight into per-stage event processing times within the DLT pipeline?

Accepted Answer

Enable the event log and query the event_log table for flow_progress events to inspect stage-level metrics such as backlog and processing time.. The event log is the built-in observability mechanism for Delta Live Tables. By querying flow_progress events, the engineer gains stage-level metrics that directly show backlog and processing time for each flow. This allows precise identification of whether ingestion or transformation is the source of latency, without deploying external monitoring tools.

Answer

Configure Databricks SQL alerts on the pipeline's target tables to detect latency by comparing row counts over time.

Answer

Monitor cluster CPU utilization in the Clusters UI and correlate spikes with pipeline run times.

Answer

Use the Spark UI's SQL tab to inspect query plans for each streaming micro-batch.

Question 6

A data engineer is using Delta Lake to manage a table that receives frequent updates and deletes. The engineer notices that query performance has degraded over time due to many small files. Which command should be used to optimize the table by compacting small files and improving query performance?

Accepted Answer

OPTIMIZE table_name. The OPTIMIZE command is designed to compact small files into larger ones, improving read performance. It can also perform Z-Ordering to cluster data. VACUUM removes old files but does not compact, ANALYZE collects statistics, and autoOptimize.optimizeWrite prevents future small files but does not fix existing ones. Therefore, OPTIMIZE is the correct command to address the current small file issue.

Answer

ANALYZE TABLE table_name COMPUTE STATISTICS

Answer

VACUUM table_name

Answer

ALTER TABLE table_name SET TBLPROPERTIES ('delta.autoOptimize.optimizeWrite' = 'true')

Question 7

A data engineer is using Auto Loader to ingest JSON files from cloud storage into a Delta table. The files contain a nested field 'address' with subfields 'city' and 'zip'. The engineer wants to flatten the nested structure during ingestion so that 'city' and 'zip' become top-level columns in the Bronze table. Which Auto Loader feature should be used to achieve this?

Accepted Answer

Apply a select transformation with col('address.city') and col('address.zip') after reading the stream.. Auto Loader ingests data with its original nested structure. To flatten nested fields into top-level columns, you must apply DataFrame transformations after reading the stream. Using select or withColumn to extract subfields like address.city and address.zip is the correct method. Auto Loader does not have a built-in flattening feature, so transformations are required.

Answer

Set cloudFiles.flattenNested to true in the Auto Loader options.

Answer

Use the cloudFiles.schemaEvolutionMode set to 'addNewColumns' to automatically flatten nested fields.

Answer

Set cloudFiles.schemaHints to specify the nested fields as top-level columns.

Question 8

A data engineer is ingesting streaming data from Apache Kafka into a Delta table using Databricks Structured Streaming. The engineer wants to ensure exactly-once processing and handle late-arriving data. Which combination of features should the engineer use?

Accepted Answer

Enable checkpointing and use watermarks with a time-based window.. Exactly-once processing in Structured Streaming requires reliable checkpointing to track offsets and state. Watermarks are used to handle late-arriving data by defining a threshold beyond which data is dropped, and they enable state cleanup for windowed operations. Combining checkpointing with watermarks provides both exactly-once semantics and late data handling, making it the correct approach for ingesting Kafka data into Delta.

Answer

Set the Kafka consumer group ID to a unique value and use foreachBatch to write to Delta.

Answer

Configure the Delta table with mergeSchema enabled and use append mode for writes.

Answer

Use the Kafka source with startingOffsets set to earliest and enable auto.offset.reset to latest.

Question 9

A data engineer is troubleshooting a Delta Live Tables pipeline that intermittently fails with 'StreamingQueryException: Job aborted due to stage failure'. The pipeline processes streaming data from a Kafka source. Which monitoring approach will best help identify the root cause of these intermittent failures?

Accepted Answer

Enable and analyze the Delta Live Tables event log for detailed error messages and stack traces.. The Delta Live Tables event log is the centralized source for pipeline run details, including error messages and stack traces. For intermittent streaming failures, it offers the most direct and detailed diagnostic information. Other monitoring tools either focus on resource usage or lack the necessary error context, making them less effective for root cause analysis in this scenario.

Answer

Set up a Databricks SQL alert on the target Delta table's row count to detect missing data.

Answer

Use the Spark UI to inspect the DAG and stage details for each failed job run.

Answer

Monitor the cluster's CPU and memory utilization metrics in the Databricks workspace.

Question 10

A data engineer is using Databricks Repos to manage a project. They need to ensure that the production job always uses the code from the 'main' branch, even if developers push changes to other branches. Which Git reference should be used in the job configuration to achieve this?

Accepted Answer

The branch name 'main'. In Databricks Repos, job configurations can reference a Git branch, tag, or commit. To always use the latest code from the 'main' branch, the job should be configured with the branch name 'main'. At runtime, Databricks checks out the latest commit on that branch. Using a commit hash or tag would pin to a specific version and not automatically update, which is unsuitable for continuous deployment.

Answer

A specific commit hash

Answer

The remote URL of the repository

Answer

A tag named 'production'

Question 11

A data engineer is deploying a Databricks job that uses a Python wheel task. The job fails with the error: 'ModuleNotFoundError: No module named 'my_library''. The wheel file is stored in DBFS at 'dbfs:/FileStore/wheels/my_library-0.1.0-py3-none-any.whl'. The job cluster is configured with a cluster policy that restricts library installation from DBFS. What is the most likely cause of the failure?

Accepted Answer

The cluster policy prevents installing libraries from DBFS, so the wheel was not installed.. The error indicates that the Python module 'my_library' is not available in the job cluster's environment. Since the wheel is stored in DBFS and the cluster policy restricts library installation from DBFS, the wheel was not installed. The engineer must either modify the cluster policy to allow DBFS libraries or relocate the wheel to a permitted source, such as a Unity Catalog volume or cloud storage, and then install it.

Answer

The job cluster does not have the 'pip' package manager installed.

Answer

The wheel file path is incorrect; it should be 'dbfs:/FileStore/wheels/my_library-0.1.0-py3-none-any.whl' without the 'dbfs:/' prefix.

Answer

The wheel file is not compatible with the cluster's Python version.

Question 12

A data engineer wants to monitor the health of a Delta Live Tables pipeline and receive alerts when the pipeline fails to meet its data quality expectations. The pipeline has several expectations defined. Which Databricks feature should the engineer use to set up these alerts?

Accepted Answer

Delta Live Tables event log with Databricks SQL alerts.. The Delta Live Tables event log captures detailed data quality metrics and expectation results. By using Databricks SQL alerts to query this log, engineers can set up precise alerts for data quality violations. Other options either lack the necessary data quality context or only provide high-level failure notifications.

Answer

Job notifications configured on the DLT pipeline's underlying job.

Answer

Databricks SQL dashboard with a query on the target Delta table.

Answer

Cluster metrics in the Databricks workspace.

Question 13

A data engineer is troubleshooting a Databricks job that fails with a 'TaskFailed' error. The job uses a cluster with autoscaling enabled. The engineer suspects that the failure is due to memory issues on the workers. Which TWO actions should the engineer take to diagnose and resolve the issue? (Choose two.)

Accepted Answer

Check the cluster's event log for 'ExecutorLostFailure' events indicating out-of-memory errors.. To diagnose memory issues, the engineer should use the Spark UI to examine memory-related metrics such as garbage collection and spill, and check the cluster event log for executor out-of-memory errors. These steps confirm whether memory is the bottleneck. Once confirmed, the engineer can adjust cluster configuration, such as increasing executor memory or optimizing the job. Disabling autoscaling or increasing driver memory does not address worker memory problems, and increasing partitions is a potential fix but not a primary diagnostic step.

Answer

Increase the driver node's memory to handle the failing tasks.

Answer

Increase the number of partitions when reading the data to reduce per-task memory usage.

Answer

Disable autoscaling to ensure the cluster has a fixed number of workers.

Question 14

A data engineer is troubleshooting a production Databricks job that intermittently fails with 'SparkOutOfMemoryError'. The job processes large datasets with skewed partitions. The engineer wants to monitor the job to proactively detect memory pressure before failures occur. Which metric should the engineer monitor on the driver and executor nodes?

Accepted Answer

JVM heap memory usage. JVM heap memory usage is the most direct indicator of memory pressure on Spark driver and executor nodes. By monitoring heap usage, engineers can detect when memory is approaching limits and intervene before OutOfMemoryError occurs. Other metrics like disk I/O, network throughput, and CPU utilization provide complementary information but do not directly measure memory consumption.

Answer

Disk I/O read/write throughput

Answer

CPU utilization percentage

Answer

Network throughput between executors

Question 15

A data engineer is troubleshooting a Databricks job that fails with a `SparkException: Job aborted due to stage failure` and the error log shows `java.lang.OutOfMemoryError: GC overhead limit exceeded` on an executor. The job processes a large dataset using a `groupByKey` operation. Which action should the engineer take to resolve the issue while minimizing changes to the existing code?

Accepted Answer

Replace `groupByKey` with `reduceByKey` to reduce memory pressure by combining values locally before shuffling.. The `groupByKey` operation shuffles all values for each key without local aggregation, which can lead to excessive memory consumption and GC overhead. Replacing it with `reduceByKey` enables map-side combining, significantly reducing the shuffle size and memory footprint. This is the most direct and code-minimal fix. Other options either mask the symptom or do not address the root cause.

Answer

Increase the executor memory by adding `spark.executor.memory=16g` to the cluster Spark configuration.

Answer

Set `spark.sql.shuffle.partitions` to a higher value to increase the number of partitions after shuffle.

Answer

Switch the job to use a larger cluster with more worker nodes to distribute the load.