Databricks-DE-Assoc Practice Questions 2 — 50 Questions

Question 1

A data engineer is designing a Bronze-to-Silver transformation pipeline using Delta Lake. They need to ensure that the Silver table contains only records where the 'transaction_id' is not null and the 'amount' is positive. Which technique best ensures data quality at this stage?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use a filter transformation in the Spark Structured Streaming query before writing to the target table.. Implementing Delta Lake expectations or inline 'where' clauses during the write process is critical for maintaining high-quality Silver layers. By filtering records before they are committed, you prevent corrupt data from propagating to downstream analytical tables. This pattern is fundamental to the Medallion Architecture, ensuring the Silver layer serves as a reliable, cleaned source of truth for downstream consumption and complex modeling tasks.

Answer

Apply a post-load DELETE statement to the Silver table after every micro-batch.

Answer

Define a separate table for rejected records and use manual SQL queries to move them later.

Answer

Set the table property 'delta.constraints.check' to filter incoming null values.

Question 2

A data engineering team is migrating a legacy data warehouse to Databricks. They want to ensure that their raw data is ingested into a 'Bronze' table in its original format. Which approach is most recommended for this ingestion layer?

Accepted Answer

Use Auto Loader with cloudFiles format. Using Auto Loader is the best practice for ingesting raw data from cloud object storage. It provides efficient, scalable, and incremental ingestion using cloud-native file notifications. By capturing the file as-is and adding metadata columns (like _rescued_data), the Bronze layer preserves the original record integrity. This ensures that if business logic changes, the team can reprocess the raw data from scratch, which is a foundational principle of the Medallion architecture.

Answer

Use a standard Spark read with schema inference enabled

Answer

Perform a manual copy of files to a local cluster before processing

Answer

Load the data into a Pandas DataFrame and write it to Delta

Question 3

A data engineering team wants to implement Git integration for their Databricks notebooks. Which workflow is considered the best practice for CI/CD in Databricks Repos?

Accepted Answer

Develop code in feature branches, merge via pull requests, and use Databricks Repos to sync production.. Integrating Databricks Repos with a Git provider like GitHub enables version control at the notebook level. This workflow allows teams to use feature branches for development, submit pull requests for code review, and merge into a main branch that triggers automated deployments. Adopting this standardizes the development lifecycle, ensures code traceability, and prevents manual, error-prone deployments in production environments, which is essential for maintaining production-grade data pipelines.

Answer

Export notebooks manually to local machines and commit them via Git CLI.

Answer

Use the Databricks REST API to push code updates directly to production notebooks.

Answer

Develop code in the production workspace and use Git only for final snapshots.

Question 4

A data engineer wants to use 'Expectations' in Delta Live Tables to monitor data quality. What happens if a record violates an expectation defined with the 'fail' constraint?

Accepted Answer

The pipeline execution stops immediately, and the batch is not committed.. In DLT, the 'fail' expectation is a strict data quality gate. If any record in a micro-batch fails the validation, the entire pipeline will stop processing, and the current batch will fail. This prevents bad data from reaching the target, which is essential for pipelines where data integrity is the highest priority. It forces the developer to address the underlying data quality issues before the pipeline can continue.

Answer

The record is automatically moved to a side table for manual review.

Answer

The record is dropped, and the pipeline continues to process the next record.

Answer

The error is logged, and the pipeline continues to process the batch.

Question 5

A data engineer needs to troubleshoot a job that is failing during the 'shuffle' phase. Which Spark UI tab should the engineer examine to analyze the shuffle partitions and identify potential imbalances?

Accepted Answer

The 'Stages' tab.. The 'Stages' tab in the Spark UI is the primary place to analyze shuffle performance. Within this tab, an engineer can view the 'Shuffle Read' and 'Shuffle Write' metrics for each task. By examining the partition-level distribution, the engineer can detect if the data is being shuffled unevenly, which is a common cause of stage-level bottlenecks and job failures during complex transformations like joins or window functions.

Answer

The 'Executors' tab.

Answer

The 'SQL' tab.

Answer

The 'Environment' tab.

Question 6

A data engineer is designing a pipeline and needs to ensure that data remains consistent during concurrent read and write operations. Which Databricks feature provides the mechanism to track and validate these operations?

Accepted Answer

Delta Lake Transaction Log. The Delta Lake transaction log, often referred to as the _delta_log, is a centralized record of all changes made to a table. It ensures atomicity and durability by serializing transactions. This mechanism is critical because it allows Databricks to manage concurrent access without locks, providing a consistent view of the data even when multiple users or processes are interacting with the same underlying storage files simultaneously.

Answer

Unity Catalog

Answer

Databricks SQL

Answer

Cluster Manager

Question 7

Which THREE of the following are primary components of the Databricks Lakehouse architecture?

Accepted Answer

Delta Lake. The Databricks Lakehouse architecture combines the best of data warehouses and data lakes by using Delta Lake for reliable storage, Spark for performant compute, and Unity Catalog for centralized governance. These components work together to provide a unified platform that supports diverse workloads, including batch processing, streaming, data science, and business intelligence, ensuring that data is governed, performant, and reliable for all organizational stakeholders.

Answer

Direct Hardware Access

Answer

Proprietary File System Format

Question 8

You are tasked with handling late-arriving data in a streaming pipeline that performs windowed aggregations. Which approach ensures that the output remains accurate while balancing memory usage?

Accepted Answer

Implement a watermark on the timestamp column and aggregate over a window.. Watermarking is the primary mechanism to handle late-arriving data. It allows the system to define a threshold for how long state should be maintained for late updates. By setting an appropriate watermark duration, you balance the requirement for accuracy (including late data) with the need for memory efficiency (cleaning up expired state). This is a foundational concept for robust streaming ETL on Databricks.

Answer

Increase the state TTL to infinity to ensure no data is ever dropped.

Answer

Use the 'complete' output mode to ensure all historical data is reprocessed.

Answer

Filter out records with timestamps older than the current processing time.

Question 9

Why is it important to use Service Principals instead of Personal Access Tokens (PATs) for CI/CD automation in Databricks?

Accepted Answer

Service Principals are not tied to an individual's lifecycle.. Service Principals provide a secure and stable identity for automated systems, independent of individual user accounts. Because PATs are tied to a specific user, they become invalid when that user leaves the company or their access is revoked. Service Principals ensure continuous operation of CI/CD pipelines, simplify security auditing, and follow the principle of least privilege, which is crucial for enterprise-grade automation and governance.

Answer

Service Principals are faster to execute than personal tokens.

Answer

Service Principals do not require any permission configuration.

Answer

Service Principals allow anyone in the organization to run pipelines.

Question 10

When promoting code from a development workspace to a production workspace, what is the primary risk of using manual notebook exports?

Accepted Answer

The process creates an inconsistent state between development and production.. Manual notebook exports lack version integrity and metadata synchronization, leading to 'code drift' where the production code doesn't match the source of truth in Git. This manual process is prone to human error, such as importing the wrong version or missing library dependencies. Standardizing on Git-integrated workflows or Databricks Asset Bundles mitigates this risk by automating the deployment and ensuring that exactly what was tested in development reaches production.

Answer

The notebooks will run significantly slower in the production environment.

Answer

The Databricks workspace will automatically delete the old notebooks.

Answer

Manual exports are prohibited by Databricks security policies.

Question 11

Refer to the exhibit. An engineer applies these configurations to a cluster. What is the primary benefit of enabling the Databricks IO Cache for a workload that involves repeatedly reading the same Delta tables?

Accepted Answer

It speeds up repeated reads by caching data on local SSDs.. The Databricks IO Cache (also known as the disk cache) accelerates data reads by caching remote data on the local SSDs of the worker nodes. For workloads that frequently query the same tables, this eliminates the latency and network overhead of repeatedly fetching data from cloud object storage. This is particularly effective for read-heavy analytical workloads, enabling significantly faster query execution times by leveraging high-speed local disk I/O.

Answer

It enables ACID transactions for non-Delta tables.

Answer

It automatically scales the cluster based on disk usage.

Answer

It forces the cluster to store all data in memory.

Question 12

A data engineer is analyzing a Spark job that is failing with 'Out of Memory' (OOM) errors. Which configuration parameter should be tuned to increase the amount of memory allocated to the execution of joins and aggregations?

Accepted Answer

spark.memory.fraction. Spark's memory management divides heap memory into storage and execution. Joins and aggregations occur in the execution memory. When these operations process large datasets that exceed the allocated execution memory, Spark may fail with OOM. Tuning the memory fraction configuration allows the engineer to shift the balance between storage (for caching) and execution, providing more headroom for complex shuffle-heavy operations like joins and group-by aggregations.

Answer

spark.executor.cores

Answer

spark.sql.shuffle.partitions

Answer

spark.driver.memory

Question 13

Which THREE of the following are benefits of using Unity Catalog over the legacy Hive Metastore?

Accepted Answer

Support for data lineage at the column level.. Unity Catalog provides a unified, centralized governance layer that spans all workspaces, clouds, and regions. Unlike the legacy Hive Metastore, which is often siloed and lacks granular control, Unity Catalog offers robust auditing, data lineage, and simplified access management. These features are vital for modern enterprises managing complex, multi-workspace environments that require consistent security postures and comprehensive visibility into data movement and usage patterns.

Answer

Automated data replication to other cloud vendors.

Answer

Automatic hardware upgrades for all clusters.

Question 14

Which of the following is a fundamental principle of implementing effective CI/CD for Databricks workflows?

Accepted Answer

Using environment variables to manage service endpoints and authentication secrets.. Effective CI/CD requires separating code logic from environment-specific configurations. By parameterizing job settings and using environment-specific variables, teams ensure that the same code can be deployed safely across development, testing, and production. This practice minimizes errors during promotion and allows for automated testing cycles, which are essential for maintaining a robust, stable, and scalable data platform in any enterprise Databricks environment.

Answer

Deploying code from a local laptop directly to the production cluster.

Answer

Merging all development code into the main branch every few months.

Answer

Ignoring unit testing until the code is fully deployed in production.

Question 15

A data engineer is tasked with securing sensitive PII data in Unity Catalog. Which THREE actions are recommended to ensure robust security and compliance?

Accepted Answer

Apply Dynamic Data Masking to columns containing PII.. Implementing a defense-in-depth strategy is crucial for PII. Using Unity Catalog's fine-grained access controls, dynamic masking, and audit logs provides a layered approach to security. These tools allow organizations to restrict access, obscure sensitive values, and maintain a verifiable trail of who accessed what data, which is essential for meeting regulatory requirements and minimizing the risk of unauthorized data breaches.

Answer

Grant SELECT access to all users to ensure data accessibility.

Answer

Store all PII data in the root metastore directory for easy access.

Question 16

A streaming job using Structured Streaming is lagging significantly behind the 'current time'. Which THREE of the following could be the root cause of this processing latency?

Accepted Answer

The trigger interval is set to a time shorter than the processing time of the batch.. Streaming latency often stems from resource contention, inefficient state management, or micro-batch trigger configurations. When the input rate exceeds the processing rate, lag accumulates. Identifying these bottlenecks requires analyzing the Spark UI for processing time vs. batch time. Understanding these factors is vital for maintaining low-latency pipelines and ensuring that SLAs are met in production-grade streaming environments deployed on Databricks.

Answer

Using a fixed-size cluster with no auto-scaling enabled.

Answer

Using too few partitions in the input source.

Question 17

Which of the following describes the correct order of operations to configure a new external location in Unity Catalog?

Accepted Answer

Create credential, create location, grant privileges.. The process begins with creating a storage credential, which holds the cloud-specific security details (like an IAM role or service account). This credential is then used to define the external location, which points to a specific path in cloud storage. Finally, you grant the user or service the 'READ FILES' or 'WRITE FILES' privilege on the external location to permit interaction with the underlying data files within the Unity Catalog framework.

Answer

Grant privileges, create location, create credential.

Answer

Create location, grant privileges, create credential.

Answer

Create credential, grant privileges, create location.

Question 18

Refer to the exhibit. A data engineer is configuring an IAM policy for a Databricks storage credential. Which of the following is the most significant security risk in this configuration?

Accepted Answer

The policy allows access to all objects in the bucket rather than using specific prefixes.. The policy uses overly permissive resource definitions and potentially excessive actions if not scoped correctly. In production environments, adhering to the principle of least privilege is critical. Granting broad 's3:ListBucket' or 's3:GetObject' access to an entire bucket without specific prefix scoping can lead to data exfiltration risks or unauthorized access to non-production data stored in the same bucket.

Answer

The policy restricts access to only a single file.

Answer

The policy lacks an explicit 'Deny' statement for all other resources.

Answer

The policy is missing a condition to require multi-factor authentication (MFA).

Question 19

An engineer has a large table that is frequently joined with small dimension tables. To optimize this, which optimization technique should be applied to the join operation?

Accepted Answer

Use a Broadcast Hash Join.. Broadcasting small tables prevents the 'shuffle' of the larger table. By sending a copy of the dimension table to every executor, Spark can perform the join locally, significantly reducing network overhead. This is a fundamental optimization for star schemas in Databricks. Knowing when and how to force broadcast joins allows engineers to dramatically speed up analytical queries by minimizing the movement of massive datasets across the cluster network.

Answer

Implement a Shuffle Hash Join.

Answer

Increase the number of shuffle partitions.

Answer

Enable Z-Ordering on the join key.

Question 20

Which component of Databricks CI/CD is responsible for executing automated tests on code before it is merged into the main branch?

Accepted Answer

The CI/CD runner (e.g., GitHub Actions, Jenkins).. CI/CD runners or build servers (like GitHub Actions, GitLab CI, or Jenkins) are the components responsible for triggering automated testing. By running tests in an isolated environment during the pull request process, these tools ensure that only validated code is merged. This 'fail-fast' approach is critical for maintaining code quality, reducing regression risks, and providing developers with immediate feedback on their changes before they ever impact the production pipeline.

Answer

The Databricks Delta Lake storage layer.

Answer

The Unity Catalog governance framework.

Answer

The Databricks SQL Warehouse.