Databricks-DE-Assoc Practice Test 14 — 15 Questions

Question 1

A data engineering team stores its Databricks notebooks and Python files in a Git repository. They want to avoid manually copying files into the workspace and ensure that the production workspace always runs the exact code version that passed tests. Which approach should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use Databricks Repos to clone the Git repository into the production workspace and check out the tested commit.. Databricks Repos provides native Git integration, enabling teams to clone repositories and check out specific commits. By checking out the tested commit in the production workspace, the team ensures that the deployed code matches what passed tests, eliminating manual copying and reducing drift. This is a core practice for implementing CI/CD with Databricks.

Answer

Schedule a Databricks Job that runs a notebook which pulls the latest code from Git at runtime.

Answer

Mount the Git repository as a DBFS mount point and reference notebooks directly from the mount.

Answer

Export notebooks as .dbc files and import them into the production workspace using the Databricks CLI.

Question 2

A data engineer is designing an ingestion pipeline using Databricks Auto Loader to process JSON files from an S3 bucket. The pipeline must handle schema evolution and ensure that data is ingested exactly once. Which two features of Auto Loader support these requirements? (Choose two.)

Accepted Answer

Exactly-once processing with checkpointing. Auto Loader's schema inference and evolution automatically adapts to changing JSON schemas, and its checkpointing mechanism ensures exactly-once processing by tracking ingested files. These two features together meet the requirements for handling schema evolution and ensuring data is ingested exactly once.

Answer

Automatic file notification and cleanup

Answer

Built-in data quality constraints

Answer

Automatic data deduplication

Question 3

A data engineer is configuring a CI/CD pipeline for Databricks notebooks using GitHub Actions. They need to authenticate to Databricks to deploy notebooks. Which authentication method is recommended for production CI/CD pipelines?

Accepted Answer

Use a service principal with OAuth tokens generated via the Databricks CLI or REST API, storing the client ID and secret in GitHub Secrets.. For production CI/CD, authentication should use a service principal with OAuth. The client ID and secret are stored as secrets in the CI system, such as GitHub Secrets. The workflow can then use the Databricks CLI or REST API to authenticate. This approach decouples automation from individual users, supports permission scoping, and is more secure than personal access tokens.

Answer

Use the Databricks CLI with interactive login during the CI run.

Answer

Store a personal access token (PAT) in GitHub Secrets and use it in the workflow.

Answer

Embed the Databricks username and password directly in the workflow YAML file.

Question 4

A data platform team is migrating their deployment process to Databricks Asset Bundles (DABs). They already have a Python wheel task defined in a Databricks Job and a set of notebooks in a Git repository. They want the bundle deployment to be repeatable across development, staging, and production targets with different cluster sizes. Which approach should they take to parameterize the target-specific cluster configuration?

Accepted Answer

Use the bundle's databricks.yml targets block with variables and override cluster node types per target.. Databricks Asset Bundles provide a targets block in databricks.yml specifically so one bundle can deploy to multiple workspaces with environment-specific overrides. Using variables for cluster properties and overriding them per target preserves a single job definition while still allowing dev, staging, and prod to use different cluster sizes. This matches the recommended DABs workflow for environment parity and repeatable CI/CD deployments.

Answer

Store cluster configuration in a Delta table and have the job read it at runtime using spark.conf.

Answer

Define cluster settings directly in the job YAML and use separate branches for each environment.

Answer

Create a separate Databricks Job for each environment and deploy them with the same bundle.

Question 5

A data engineering team stores notebooks in a Git repository and wants automated deployments to a Databricks workspace. They configure a GitHub Actions workflow that runs a Databricks CLI command to deploy Databricks Asset Bundles. The workflow authenticates using a service principal OAuth token stored in GitHub Secrets. After the first successful run, subsequent runs fail with an authentication error. The token was created with a 1-hour lifetime. What should the team do to ensure the workflow can authenticate reliably on every run?

Accepted Answer

Configure the GitHub Actions workflow to obtain a short-lived OAuth token dynamically from the identity provider using a federated identity or client credentials flow before each deployment.. Short-lived OAuth tokens expire, so a CI/CD pipeline must fetch a new token at the start of each run. Using the client credentials flow or workload identity federation with the service principal allows the GitHub Actions workflow to authenticate dynamically without storing a long-lived secret. This is the secure, maintainable pattern for automated Databricks deployments.

Answer

Run the GitHub Actions workflow on a self-hosted runner inside the Databricks workspace network so that the CLI can use an instance profile for authentication.

Answer

Increase the service principal OAuth token lifetime to 24 hours in the Databricks account console and update the secret in GitHub.

Answer

Replace the service principal OAuth token with a personal access token (PAT) generated for a workspace admin user and store it in GitHub Secrets.

Question 6

A team is using Databricks Asset Bundles to manage a job that writes to a Unity Catalog table. The bundle is deployed to a staging workspace for testing and then to a production workspace. The team wants the job to use different catalog and schema names in each environment without duplicating the entire bundle. Which approach should they use?

Accepted Answer

Define bundle variables for the catalog and schema, and set different values for each variable in the staging and production targets within the databricks.yml file.. Databricks Asset Bundles allow a single job definition to be deployed to multiple environments by using variables that are resolved differently per target. Defining catalog and schema variables and setting their values under the staging and production targets in databricks.yml keeps the bundle DRY and ensures each workspace uses the correct Unity Catalog objects.

Answer

Store the catalog and schema names in a Databricks secret scope and reference them in the job's notebook using dbutils.secrets.get.

Answer

Use a single target and pass the catalog and schema names as job parameters at runtime through the Databricks Jobs API.

Answer

Create two separate bundle configuration files, one for staging and one for production, and manually copy the job definition into each.

Question 7

A data engineer has a Delta table named silver_events with columns event_id (string), event_ts (timestamp), and payload (string). The table is partitioned by event_date (derived from event_ts). The engineer needs to update the payload column for all events that occurred on '2024-06-01' based on a mapping table named updates (event_id, new_payload). Which PySpark operation should be used to perform this update efficiently while preserving Delta Lake ACID guarantees?

Accepted Answer

Perform a MERGE INTO operation on silver_events using updates as the source, matching on event_id and filtering both source and target to event_date = '2024-06-01'.. The correct approach is to use MERGE INTO, which is designed for upserts and updates based on a source table. Filtering both the target and source to the specific event_date partition ensures that only the relevant data is processed, optimizing performance. MERGE INTO also provides ACID guarantees, ensuring that the update is atomic and isolated, which is critical for maintaining data integrity in a Delta Lake table.

Answer

Use DataFrame.write.mode('overwrite').partitionBy('event_date').saveAsTable('silver_events') with the updated DataFrame containing only the rows for '2024-06-01'.

Answer

Read the entire silver_events table into a DataFrame, apply a join with updates, and then use DataFrame.write.mode('overwrite').saveAsTable('silver_events') to persist the changes.

Answer

Use the DeltaTable API's update method with a condition on event_date = '2024-06-01' and a join to the updates table.

Question 8

A data engineer is setting up a CI/CD pipeline that runs unit tests on transformation logic before deploying notebooks to a production Databricks workspace. The tests must run quickly and not depend on a live Databricks cluster or external data sources. Which approach best meets these requirements?

Accepted Answer

Extract transformation logic into pure Python functions and test them locally with a framework like pytest, using sample DataFrames created in memory.. Unit tests for transformation logic should be fast, isolated, and runnable without a Databricks cluster. Extracting logic into pure Python functions and testing them with pytest and in-memory DataFrames achieves this. This practice also encourages modular code that is easier to maintain and integrate into CI/CD pipelines.

Answer

Run the tests as a Databricks job on a small interactive cluster in the development workspace, using production data for realism.

Answer

Configure the pipeline to trigger a Delta Live Tables pipeline in the development workspace and check the pipeline's event log for errors.

Answer

Use Databricks Repos to run notebooks interactively and manually verify the output before merging the pull request.

Question 9

A CI/CD pipeline must run unit tests on Python transformation code before deploying a Databricks job. The tests should execute quickly without starting a cluster and should validate the transformation logic in isolation. Which approach best meets these requirements?

Accepted Answer

Use pytest in the CI runner to test the transformation functions directly, mocking Spark or using a local SparkSession only where needed.. Unit tests for transformation logic belong in the CI runner, not on a cluster. Using pytest with pure functions and a local SparkSession where necessary keeps tests fast, isolated, and free of cloud compute. This catches logic errors before deployment, whereas bundle validation only checks configuration and integration tests against production data are slow and risky.

Answer

Deploy the job to a development workspace and run an integration test against production data to confirm the transformations produce expected results.

Answer

Configure the pipeline to run databricks bundle validate and rely on schema validation to confirm the transformation logic is correct.

Answer

Run the tests as a Databricks job task on an all-purpose cluster so the tests use the same runtime as production.

Question 10

A data engineer has a Unity Catalog table named `sales.raw.orders` that contains a column `credit_card` with sensitive data. The security team requires that users in the `analyst` group see only the last four digits of the credit card number when querying this table, while all other users with appropriate privileges see the full value. The data engineer wants to implement this with minimal disruption to existing queries. Which approach should the data engineer take?

Accepted Answer

Apply a column mask function to the `credit_card` column that checks if the user is in the `analyst` group and, if so, returns only the last four digits.. Unity Catalog column masks allow dynamic data masking at query time without altering the underlying data or requiring separate views. A mask function can evaluate the current user's group memberships and return a transformed value, such as the last four digits. This meets the security requirement while preserving existing queries and access patterns.

Answer

Use a column mask function with Unity Catalog to apply row-level filtering based on the user's group membership.

Answer

Create a row filter on the table that restricts analysts to rows where the credit card number ends with certain digits, and grant them access to the table.

Answer

Create a dynamic view that applies a masking function to the `credit_card` column, and grant the `analyst` group access to the view instead of the table.

Question 11

A team wants to automate deployment of Databricks jobs from a Git repository using a CI/CD pipeline. They need a tool that reads a declarative project definition and creates or updates jobs, pipelines, and notebooks in a target workspace. Which Databricks capability should they use?

Accepted Answer

Databricks Asset Bundles, using databricks bundle deploy to apply the project definition to the target workspace.. Databricks Asset Bundles are the declarative deployment mechanism for Databricks resources. A databricks.yml file plus source files defines jobs, pipelines, and notebooks, and databricks bundle deploy applies that definition to a target workspace. Repos handle file sync for development, SQL warehouse APIs run queries, and cluster policies govern compute; none of them deploys resource definitions from Git.

Answer

Cluster policies, by attaching a policy that references the Git repository and applies job configurations at cluster start.

Answer

Databricks Repos, by adding the repository and clicking Sync in the target workspace before each release.

Answer

The Databricks SQL warehouse API, using scheduled queries to create jobs and pipelines in the target workspace.

Question 12

A data engineer is building a Lakeflow Job with a task that runs a SQL notebook. The task must run only on weekdays and must be completed before 9 AM. The engineer wants to configure the schedule to meet these requirements. What should the engineer do?

Accepted Answer

Use a cron expression with a schedule that runs at 8 AM Monday through Friday.. The correct method is to use a cron schedule that runs at 8 AM Monday through Friday. Lakeflow Jobs natively support cron expressions, allowing precise control over run times and days. This satisfies the weekday-only and before-9-AM requirements without additional complexity.

Answer

Configure the job to run continuously with a 1-hour pause between runs and enable a weekday filter.

Answer

Set the job to trigger on a file arrival event and filter for weekdays.

Answer

Use a Databricks SQL alert to trigger the job when a condition is met.

Question 13

A data engineer is building a Lakeflow Job that must process a parameterized date range. The engineer wants to pass start_date and end_date values into a notebook task at runtime and have those values available as widget-like parameters inside the notebook. Which approach should the engineer use?

Accepted Answer

Define job parameters and reference them in the notebook using the task's base parameters, then read them with dbutils.widgets.get in the notebook.. Lakeflow Jobs parameters are surfaced to notebook tasks as widget values, so the notebook can call dbutils.widgets.get with the parameter name. Defining start_date and end_date as job or task base parameters makes them available per run and visible in the job UI, enabling dynamic date-range processing without hardcoding or external lookups.

Answer

Store the dates in a Delta table and have the notebook query that table at startup.

Answer

Use the notebook's %run magic to pass arguments from a parent notebook.

Answer

Set environment variables on the job cluster and read them with os.environ in the notebook.

Question 14

A team is setting up a CI/CD pipeline that deploys Databricks Asset Bundles to production. They want the pipeline to be secure and to fail fast before any production resources are changed. Which TWO practices should they implement? (Choose two.)

Accepted Answer

Authenticate the deployment using a service principal with credentials injected from the CI system's secret manager at runtime.. Failing fast and staying secure means validating the bundle before deploying and authenticating with a service principal whose secret comes from the CI secret manager. Validation catches configuration errors before production is touched, and the secret manager keeps credentials out of the repository. Plaintext secrets, admin-level permissions, and branch-agnostic production deploys all increase risk without meeting the stated goals.

Answer

Run the deployment on every push to any branch so problems are discovered as early as possible in feature development.

Answer

Store the service principal secret as a plaintext variable in the repository's CI configuration file so all pipeline stages can read it.

Answer

Grant the service principal workspace admin rights so it can deploy any resource without encountering permission errors.

Question 15

A data engineer runs a nightly Databricks job that reads a large Delta table and writes aggregated results to another Delta table. The cluster logs show many small files in the source table, and the job runtime has increased steadily over weeks. The engineer wants to reduce the number of files without rewriting the entire table. Which command should be used?

Accepted Answer

OPTIMIZE sales. The small-file problem is a common cause of slow reads in Delta Lake. OPTIMIZE performs bin-packing to combine small files into larger ones, reducing the number of files that must be opened during a read. It can be run on the entire table or a subset using a WHERE clause, and it does not require rewriting the entire table, making it the appropriate maintenance command for this scenario.

Answer

ANALYZE TABLE sales COMPUTE STATISTICS

Answer

ALTER TABLE sales SET TBLPROPERTIES ('delta.autoOptimize.optimizeWrite' = 'true')

Answer

VACUUM sales RETAIN 0 HOURS