Databricks-DE-Assoc Implementing CI/CD Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Deploy the job to a development workspace and run an integration test against production data to confirm the transformations produce expected results.

  Why it's wrong here

  Integration tests against production data are slow, depend on external state, and risk exposing or corrupting real data. They also require a deployment before any test runs, so failures are caught late. This approach does not provide fast, isolated unit testing of transformation logic and violates the principle of testing before deployment in a CI/CD pipeline.

• ✗

  Configure the pipeline to run databricks bundle validate and rely on schema validation to confirm the transformation logic is correct.

  Why it's wrong here

  databricks bundle validate checks the bundle configuration and resource definitions, not the correctness of transformation logic. It catches YAML errors and invalid references but will not detect a wrong join or a mishandled null. Treating validation as a substitute for unit tests leaves logic bugs undetected, so it does not meet the requirement to validate transformation code in isolation.

• ✓

  Use pytest in the CI runner to test the transformation functions directly, mocking Spark or using a local SparkSession only where needed.

  Why this is correct

  pytest runs in the CI runner with no cluster, so it is fast and isolated. Transformation logic written as pure functions can be tested directly, and a local SparkSession can be created only for tests that need Spark APIs. This keeps the feedback loop short, avoids cloud compute costs, and validates logic before deployment, which is exactly what a pre-deploy unit test stage should do.

• ✗

  Run the tests as a Databricks job task on an all-purpose cluster so the tests use the same runtime as production.

  Why it's wrong here

  Running tests as a job task requires provisioning compute, which is slow and costly for every commit. It also couples test execution to the Databricks control plane, making the pipeline slower and harder to debug. Unit tests for transformation logic should run in the CI runner without a cluster, so this approach fails the speed and isolation requirements even though it uses the production runtime.