Databricks-DE-Assoc Implementing CI/CD Practice Question
A data engineer is setting up a CI/CD pipeline that runs unit tests on transformation logic before deploying notebooks to a production Databricks workspace. The tests must run quickly and not depend on a live Databricks cluster or external data sources. Which approach best meets these requirements?
⚠ Common exam trap
The trap here is equating any automated test on Databricks with a unit test, when true unit tests should avoid cluster dependencies and external data entirely.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Extract transformation logic into pure Python functions and test them locally with a framework like pytest, using sample DataFrames created in memory.
Unit tests for transformation logic should be fast, isolated, and runnable without a Databricks cluster. Extracting logic into pure Python functions and testing them with pytest and in-memory DataFrames achieves this. This practice also encourages modular code that is easier to maintain and integrate into CI/CD pipelines.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Extract transformation logic into pure Python functions and test them locally with a framework like pytest, using sample DataFrames created in memory.
Why this is correct
Extracting logic into pure Python functions allows unit tests to run locally without a cluster or external data. Using pytest with in-memory DataFrames (for example, via PySpark local mode or pandas) keeps tests fast and deterministic. This is the standard approach for testing transformation logic in a CI/CD pipeline before deployment.
- ✗
Run the tests as a Databricks job on a small interactive cluster in the development workspace, using production data for realism.
Why it's wrong here
Running tests on a live cluster with production data is slow, expensive, and introduces dependency on external systems. It also risks exposing production data to test code. The requirement is for fast, isolated tests, so a cluster-based approach with production data does not meet the stated constraints.
- ✗
Configure the pipeline to trigger a Delta Live Tables pipeline in the development workspace and check the pipeline's event log for errors.
Why it's wrong here
A Delta Live Tables pipeline run requires a cluster, consumes resources, and is not a unit test. It tests end-to-end behavior rather than isolated transformation logic. This approach is too slow and heavyweight for the stated goal of fast tests that do not depend on a live cluster.
- ✗
Use Databricks Repos to run notebooks interactively and manually verify the output before merging the pull request.
Why it's wrong here
Manual verification in Databricks Repos is not automated and does not provide repeatable, fast feedback in a CI/CD pipeline. It also depends on a human and a running workspace, which contradicts the requirement for quick, independent tests. Automated unit tests are needed to gate merges reliably.
About these practice questions
One of 276 original Databricks-DE-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Assoc exam.