Databricks-DE-Assoc Practice Test 6 — 10 Questions

Question 1

A data engineer is designing a Bronze-to-Silver transformation pipeline using Delta Lake. They need to ensure that the Silver table contains only records where the 'transaction_id' is not null and the 'amount' is positive. Which technique best ensures data quality at this stage?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a filter transformation in the Spark Structured Streaming query before writing to the target table.. Implementing Delta Lake expectations or inline 'where' clauses during the write process is critical for maintaining high-quality Silver layers. By filtering records before they are committed, you prevent corrupt data from propagating to downstream analytical tables. This pattern is fundamental to the Medallion Architecture, ensuring the Silver layer serves as a reliable, cleaned source of truth for downstream consumption and complex modeling tasks.

Answer

Apply a post-load DELETE statement to the Silver table after every micro-batch.

Answer

Define a separate table for rejected records and use manual SQL queries to move them later.

Answer

Set the table property 'delta.constraints.check' to filter incoming null values.

Question 2

A data engineering team is migrating a legacy data warehouse to Databricks. They want to ensure that their raw data is ingested into a 'Bronze' table in its original format. Which approach is most recommended for this ingestion layer?

Accepted Answer

Use Auto Loader with cloudFiles format. Using Auto Loader is the best practice for ingesting raw data from cloud object storage. It provides efficient, scalable, and incremental ingestion using cloud-native file notifications. By capturing the file as-is and adding metadata columns (like _rescued_data), the Bronze layer preserves the original record integrity. This ensures that if business logic changes, the team can reprocess the raw data from scratch, which is a foundational principle of the Medallion architecture.

Answer

Use a standard Spark read with schema inference enabled

Answer

Perform a manual copy of files to a local cluster before processing

Answer

Load the data into a Pandas DataFrame and write it to Delta

Question 3

A data engineering team wants to implement Git integration for their Databricks notebooks. Which workflow is considered the best practice for CI/CD in Databricks Repos?

Accepted Answer

Develop code in feature branches, merge via pull requests, and use Databricks Repos to sync production.. Integrating Databricks Repos with a Git provider like GitHub enables version control at the notebook level. This workflow allows teams to use feature branches for development, submit pull requests for code review, and merge into a main branch that triggers automated deployments. Adopting this standardizes the development lifecycle, ensures code traceability, and prevents manual, error-prone deployments in production environments, which is essential for maintaining production-grade data pipelines.

Answer

Export notebooks manually to local machines and commit them via Git CLI.

Answer

Use the Databricks REST API to push code updates directly to production notebooks.

Answer

Develop code in the production workspace and use Git only for final snapshots.

Question 4

A data engineer wants to use 'Expectations' in Delta Live Tables to monitor data quality. What happens if a record violates an expectation defined with the 'fail' constraint?

Accepted Answer

The pipeline execution stops immediately, and the batch is not committed.. In DLT, the 'fail' expectation is a strict data quality gate. If any record in a micro-batch fails the validation, the entire pipeline will stop processing, and the current batch will fail. This prevents bad data from reaching the target, which is essential for pipelines where data integrity is the highest priority. It forces the developer to address the underlying data quality issues before the pipeline can continue.

Answer

The record is automatically moved to a side table for manual review.

Answer

The record is dropped, and the pipeline continues to process the next record.

Answer

The error is logged, and the pipeline continues to process the batch.

Question 5

A data engineer needs to troubleshoot a job that is failing during the 'shuffle' phase. Which Spark UI tab should the engineer examine to analyze the shuffle partitions and identify potential imbalances?

Accepted Answer

The 'Stages' tab.. The 'Stages' tab in the Spark UI is the primary place to analyze shuffle performance. Within this tab, an engineer can view the 'Shuffle Read' and 'Shuffle Write' metrics for each task. By examining the partition-level distribution, the engineer can detect if the data is being shuffled unevenly, which is a common cause of stage-level bottlenecks and job failures during complex transformations like joins or window functions.

Answer

The 'Executors' tab.

Answer

The 'SQL' tab.

Answer

The 'Environment' tab.

Question 6

A data engineer is designing a pipeline and needs to ensure that data remains consistent during concurrent read and write operations. Which Databricks feature provides the mechanism to track and validate these operations?

Accepted Answer

Delta Lake Transaction Log. The Delta Lake transaction log, often referred to as the _delta_log, is a centralized record of all changes made to a table. It ensures atomicity and durability by serializing transactions. This mechanism is critical because it allows Databricks to manage concurrent access without locks, providing a consistent view of the data even when multiple users or processes are interacting with the same underlying storage files simultaneously.

Answer

Unity Catalog

Answer

Databricks SQL

Answer

Cluster Manager

Question 7

Which THREE of the following are primary components of the Databricks Lakehouse architecture?

Accepted Answer

Delta Lake. The Databricks Lakehouse architecture combines the best of data warehouses and data lakes by using Delta Lake for reliable storage, Spark for performant compute, and Unity Catalog for centralized governance. These components work together to provide a unified platform that supports diverse workloads, including batch processing, streaming, data science, and business intelligence, ensuring that data is governed, performant, and reliable for all organizational stakeholders.

Answer

Direct Hardware Access

Answer

Proprietary File System Format

Question 8

You are tasked with handling late-arriving data in a streaming pipeline that performs windowed aggregations. Which approach ensures that the output remains accurate while balancing memory usage?

Accepted Answer

Implement a watermark on the timestamp column and aggregate over a window.. Watermarking is the primary mechanism to handle late-arriving data. It allows the system to define a threshold for how long state should be maintained for late updates. By setting an appropriate watermark duration, you balance the requirement for accuracy (including late data) with the need for memory efficiency (cleaning up expired state). This is a foundational concept for robust streaming ETL on Databricks.

Answer

Increase the state TTL to infinity to ensure no data is ever dropped.

Answer

Use the 'complete' output mode to ensure all historical data is reprocessed.

Answer

Filter out records with timestamps older than the current processing time.

Question 9

Why is it important to use Service Principals instead of Personal Access Tokens (PATs) for CI/CD automation in Databricks?

Accepted Answer

Service Principals are not tied to an individual's lifecycle.. Service Principals provide a secure and stable identity for automated systems, independent of individual user accounts. Because PATs are tied to a specific user, they become invalid when that user leaves the company or their access is revoked. Service Principals ensure continuous operation of CI/CD pipelines, simplify security auditing, and follow the principle of least privilege, which is crucial for enterprise-grade automation and governance.

Answer

Service Principals are faster to execute than personal tokens.

Answer

Service Principals do not require any permission configuration.

Answer

Service Principals allow anyone in the organization to run pipelines.

Question 10

When promoting code from a development workspace to a production workspace, what is the primary risk of using manual notebook exports?

Accepted Answer

The process creates an inconsistent state between development and production.. Manual notebook exports lack version integrity and metadata synchronization, leading to 'code drift' where the production code doesn't match the source of truth in Git. This manual process is prone to human error, such as importing the wrong version or missing library dependencies. Standardizing on Git-integrated workflows or Databricks Asset Bundles mitigates this risk by automating the deployment and ensuring that exactly what was tested in development reaches production.

Answer

The notebooks will run significantly slower in the production environment.

Answer

The Databricks workspace will automatically delete the old notebooks.

Answer

Manual exports are prohibited by Databricks security policies.