Databricks-DE-Assoc Practice Test 18 — 15 Questions

Question 1

When ingesting data from cloud storage, what is the most important reason to use a Service Principal or IAM Role instead of individual user credentials?

Accepted Answer

To ensure the pipeline continues to run if the user leaves the company.. Using service-level authentication is a security best practice that ensures ingestion pipelines are not tied to specific individuals. This prevents production jobs from failing when a user leaves the organization or has their permissions revoked. It also provides a more secure and auditable way to manage access to sensitive data in cloud environments.

Answer

To increase the maximum file size that can be ingested by Spark.

Answer

To enable the use of the COPY INTO command in SQL warehouses.

Answer

To allow Auto Loader to use directory listing instead of notifications.

Question 2

Which security feature should a data engineer configure to ensure that audit logs from all Databricks workspaces are captured and stored in a single, centralized location?

Accepted Answer

Diagnostic logs.. Databricks diagnostic logs are the primary mechanism for collecting security, governance, and operational audit information. By configuring diagnostic log streaming to a centralized location, such as a cloud-native log repository (e.g., Azure Log Analytics or AWS S3), organizations can perform centralized monitoring, anomaly detection, and compliance reporting across their entire Databricks estate, ensuring no audit trail is lost or siloed in individual workspaces.

Answer

Cluster event logs.

Answer

System schema (system.access).

Answer

Workspace-local storage.

Question 3

A data engineer is using Auto Loader to ingest data from a Kafka topic into a Delta table. The engineer wants to ensure that the ingestion handles late-arriving data and provides exactly-once semantics. Which combination of features should the engineer use?

Accepted Answer

Structured Streaming with Kafka source, watermarking, and checkpointing. For Kafka ingestion with exactly-once semantics and late data handling, Structured Streaming with the Kafka source, watermarking, and checkpointing is the correct approach. Checkpointing ensures exactly-once by tracking offsets, and watermarking allows processing of late-arriving data up to a specified threshold.

Answer

Auto Loader with cloudFiles.useNotifications = true and checkpointing

Answer

COPY INTO with Kafka connector and FORCE = true

Answer

Delta Live Tables with Kafka source and APPLY CHANGES

Question 4

A data engineer needs to store structured data in a cloud object storage location while maintaining full ACID guarantees. Which storage format is the foundation of the Databricks Lakehouse architecture that enables this functionality?

Accepted Answer

Delta Lake. Delta Lake is the open-source storage layer that brings reliability and ACID transactions to data lakes. By utilizing transaction logs and Parquet files, it ensures data consistency for concurrent read and write operations. Understanding Delta Lake is fundamental for Databricks engineers, as it underpins virtually all data operations within the platform, enabling features like time travel, schema enforcement, and efficient upserts on massive datasets.

Answer

JSON

Answer

CSV

Answer

Avro

Question 5

Refer to the exhibit. A data engineer is reviewing the configuration of a Delta table that is frequently queried by 'customer_id'. Given the current metadata, which action will provide the most significant improvement to query performance?

Accepted Answer

Run OPTIMIZE with ZORDER BY (customer_id). The table is currently suffering from poor data layout due to the small file size and lack of clustering on the frequently queried 'customer_id' column. Performing an OPTIMIZE operation with Z-Ordering on 'customer_id' will cluster the data physically on disk based on that key. This enables the engine to perform data skipping, reducing the amount of data scanned and significantly speeding up point lookups and aggregations by customer.

Answer

Execute VACUUM to remove old files

Answer

Re-partition the table by 'customer_id'

Answer

Increase the 'order_date' partition size

Question 6

A data engineer needs to pass the execution date to a job task dynamically. Which feature should they use?

Accepted Answer

Defining job parameters in the task configuration.. Using Job Parameters allows engineers to inject dynamic values into tasks at runtime. This capability is crucial for backfilling data or running incremental pipelines where the processing logic depends on specific dates or data partitions. By parameterizing tasks, engineers create reusable jobs that can handle varying datasets without requiring manual code changes, thereby improving the maintainability and scalability of the entire data pipeline architecture.

Answer

Hardcoding the date in the notebook.

Answer

Creating a new cluster for each date.

Answer

Using global workspace variables.

Question 7

Refer to the exhibit. A data engineer attempts to run the GRANT statement above in a Databricks SQL query editor. What is the most likely reason for failure?

Accepted Answer

The table must be referenced using a three-level namespace.. The command fails because Unity Catalog requires the full three-level namespace (catalog.schema.table). Without specifying 'main.prod.sales_data', the engine cannot locate the table within the hierarchical metadata store. This strict requirement ensures that all data access is explicitly scoped, preventing ambiguity in multi-catalog or multi-schema environments, which is a fundamental tenet of Unity Catalog's governance model for enterprise-grade security and asset management.

Answer

The syntax of the GRANT statement is incorrect.

Answer

The table does not exist.

Answer

The group 'data_scientists' does not exist.

Question 8

A data engineer is working with a large, partitioned table and needs to perform a complex transformation. Which THREE of the following strategies will optimize query performance for this transformation?

Accepted Answer

Use Z-Ordering on columns frequently used in WHERE clauses.. Performance optimization in Databricks relies on intelligent data layout and compute utilization. Techniques like Z-Ordering, proper partitioning, and effective filtering are essential for reducing the amount of data scanned. By selecting the right combinations of these strategies, engineers can significantly reduce the I/O overhead and compute costs, which is paramount for large-scale production ETL/ELT workflows.

Answer

Partition the table by every column in the dataset.

Answer

Always use a full table scan to ensure data integrity.

Question 9

A data engineer needs to create a Silver Delta table that contains only distinct, non-null `customer_id` values from a Bronze table, and the result must be refreshed idempotently each night. Which statement best satisfies the requirement?

Accepted Answer

CREATE OR REPLACE TABLE silver AS SELECT DISTINCT customer_id FROM bronze WHERE customer_id IS NOT NULL. CREATE OR REPLACE TABLE with a DISTINCT and null-filtered SELECT atomically rebuilds the Silver table each night, guaranteeing the same result on every run. This satisfies both the distinct non-null requirement and idempotent refresh. Appending via INSERT INTO, incremental MERGE, or relying on a pre-existing target for INSERT OVERWRITE does not meet all the stated conditions.

Answer

INSERT OVERWRITE silver SELECT DISTINCT customer_id FROM bronze WHERE customer_id IS NOT NULL

Answer

MERGE INTO silver AS t USING bronze AS s ON t.customer_id = s.customer_id WHEN NOT MATCHED THEN INSERT (customer_id) VALUES (s.customer_id)

Answer

INSERT INTO silver SELECT DISTINCT customer_id FROM bronze WHERE customer_id IS NOT NULL

Question 10

Refer to the exhibit. A user who is a member of 'data_analysts_group' reports they cannot see the 'orders' table in the Catalog Explorer. What is the most likely cause?

Accepted Answer

The user lacks the 'USAGE' privilege on the parent objects.. In Unity Catalog, the 'USAGE' privilege is a mandatory prerequisite for navigating any parent object. Even if the user has SELECT access on the table, they must also hold USAGE on the parent catalog and schema to traverse the hierarchy and discover the table. Without these permissions, the object remains invisible to the user in the UI, even if they have technical access to the underlying data records.

Answer

The user lacks the 'SELECT' privilege.

Answer

The table is not registered in the Hive Metastore.

Answer

The group does not exist in the workspace.

Question 11

What is the primary purpose of a 'feature branch' in a Git-based workflow for Databricks?

Accepted Answer

To isolate changes so they can be tested before merging to production.. Feature branches allow developers to build and test new functionality in isolation, without affecting the main production code. This workflow is essential for CI/CD because it facilitates peer reviews, automated integration testing, and safe integration. By keeping the main branch stable and only merging validated features, teams maintain a high standard of quality and avoid breaking the production environment with incomplete or untested code changes, which is a core benefit of modern DevOps practices.

Answer

To allow multiple users to edit the same file simultaneously without sync.

Answer

To bypass the authentication requirements for the workspace.

Answer

To automatically deploy changes to the production environment.

Question 12

A data engineer notices that a scheduled Delta Lake maintenance pipeline is running significantly slower than expected. Upon checking the table history, they see that hundreds of tiny, fragmented data files have accumulated due to frequent streaming micro-batches. Which specific optimization command should the data engineer run first to resolve this file-size bottleneck?

Accepted Answer

Invoke the OPTIMIZE table_name command to compact small files into larger, optimized data files and improve subsequent query execution efficiency.. Running OPTIMIZE reorganizes the layout of Delta Lake data by compacting small files into larger, uniform files of roughly 1 GB in size. This significantly reduces metadata overhead and improves scan performance for subsequent read queries. While VACUUM removes old physical files, it does not compact active files. OPTIMIZE directly addresses the core symptom of small file proliferation caused by frequent streaming micro-batches.

Answer

Run VACUUM table_name RETAIN 0 HOURS to instantly purge all historical file references and force file compaction across the entire dataset.

Answer

Execute REFRESH TABLE table_name to clear the local metastore cache and force Spark to rebuild the underlying file index from scratch.

Answer

Execute ALTER TABLE table_name SET TBLPROPERTIES (delta.autoOptimize.optimizeWrite = true) to rewrite past historical micro-batches retroactively.

Question 13

A data engineer is creating a Lakeflow Job that must run a Python script stored in DBFS. The engineer wants to ensure the script is executed with the correct dependencies and environment. Which task type should be used?

Accepted Answer

Python script task. Lakeflow Jobs offer a dedicated Python script task type that runs a Python file from DBFS or other storage locations. It supports specifying libraries and environment settings, ensuring the script runs with the correct dependencies. Notebook, JAR, and pipeline tasks serve different purposes and cannot directly execute a standalone Python script with its own environment.

Answer

JAR task

Answer

Notebook task

Answer

Pipeline task

Question 14

A data engineer is using Auto Loader to stream data from Kafka into a Delta table. The Kafka topic receives messages in Avro format, and the schema is stored in a Confluent Schema Registry. The engineer wants Auto Loader to automatically fetch the schema from the registry and evolve it as new versions are registered. Which configuration should the engineer use?

Accepted Answer

Auto Loader cannot ingest from Kafka; use Structured Streaming with the Kafka source and Schema Registry integration.. Auto Loader is a file ingestion tool and does not support Kafka as a source. For Kafka ingestion with Avro and Schema Registry, the appropriate method is to use Spark Structured Streaming's Kafka source, which can integrate with Schema Registry to fetch and evolve schemas. The engineer should not attempt to use Auto Loader for this purpose. Therefore, the correct answer is to use Structured Streaming with the Kafka source and the necessary Schema Registry configurations.

Answer

Use the Kafka source with the option kafka.schema.registry.url and set the value format to 'avro'.

Answer

Set the option cloudFiles.format to 'avro' and provide the schema registry URL via cloudFiles.schemaRegistryUrl.

Answer

Set the option cloudFiles.useNotifications to true and configure Kafka as a notification source.

Question 15

Refer to the exhibit. A job fails with the provided error message. What is the most likely cause of this failure in a Databricks Delta Lake environment?

Accepted Answer

The VACUUM retention threshold was set too low.. This error occurs when the Delta log entries point to files that no longer exist, typically caused by a VACUUM operation running with a retention period shorter than the time required for long-running streaming jobs to finish. When a file is physically deleted by VACUUM, but a reader still references it via the transaction log, the read operation fails because the file object is inaccessible in the underlying storage.

Answer

The Delta table has reached the maximum file limit.

Answer

The cluster running the job lacks proper IAM permissions.

Answer

The transaction log is corrupted and requires manual deletion.