Databricks-DE-Assoc Practice Test 13 — 15 Questions

Question 1

A data engineer is working in a Databricks workspace where Unity Catalog is enabled. They need to run a SQL query that reads from the table sales in the catalog prod and schema marketing. Which fully qualified name should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

prod.marketing.sales. Unity Catalog uses a three-level namespace: catalog.schema.table. For a table named sales in the marketing schema of the prod catalog, the fully qualified name is prod.marketing.sales. This naming convention allows objects to be referenced unambiguously across the workspace and is fundamental to Unity Catalog governance.

Answer

prod.sales.marketing

Answer

marketing.prod.sales

Answer

sales.marketing.prod

Question 2

A data engineer is configuring a Databricks cluster to run a Spark job that processes large datasets. The job requires high memory and will run for several hours. The engineer wants to minimize costs while ensuring the job completes successfully. Which cluster configuration should the engineer choose?

Accepted Answer

Use a job cluster with autoscaling enabled, high-memory instance types, and a minimum of 2 workers and a maximum of 10 workers.. For long-running batch jobs, job clusters are more cost-effective than all-purpose clusters. Autoscaling allows the cluster to scale up or down based on workload, ensuring resources are used efficiently. High-memory instances meet the memory requirement. Setting a minimum and maximum worker count provides boundaries for autoscaling, preventing over-provisioning. This combination minimizes cost while ensuring the job has sufficient resources.

Answer

Use an all-purpose cluster with a fixed number of high-memory workers and enable autoscaling to reduce costs during peak hours.

Answer

Use a job cluster with a fixed number of high-memory workers and enable auto-termination after 30 minutes of inactivity.

Answer

Use an all-purpose cluster with autoscaling enabled and a minimum of 2 workers and a maximum of 10 workers, with high-memory instance types.

Question 3

A data engineer needs to run a Databricks notebook on a schedule every day at 8:00 AM UTC. The notebook performs data transformations and writes results to a Delta table. The engineer wants to ensure the job runs reliably and can be monitored. Which Databricks feature should the engineer use to schedule and monitor the notebook?

Accepted Answer

Databricks Jobs with a cron schedule.. Databricks Jobs is the correct feature for scheduling and monitoring notebooks. It supports cron-based schedules, provides a UI for monitoring runs, and can send alerts on failure. Other options like SQL Alerts, Repos, or Delta Live Tables serve different purposes and do not provide the required scheduling and monitoring for a notebook. Therefore, Databricks Jobs is the appropriate choice.

Answer

Databricks Delta Live Tables with a continuous pipeline.

Answer

Databricks Repos with a CI/CD pipeline.

Answer

Databricks SQL Alerts.

Question 4

A data engineer is using Databricks Auto Loader to stream CSV files from an ADLS Gen2 container into a Delta table. The source directory contains a mix of files, but only files with the prefix 'sales_' should be ingested. The engineer wants Auto Loader to ignore all other files without moving or deleting them. Which Auto Loader option should the engineer configure to achieve this?

Accepted Answer

cloudFiles.pathGlobFilter. Auto Loader provides the cloudFiles.pathGlobFilter option to filter ingested files using glob patterns. By setting it to 'sales_*', only files with the desired prefix are processed, and others are ignored without being moved or deleted. This directly satisfies the requirement to ingest a subset of files based on filename.

Answer

cloudFiles.format

Answer

cloudFiles.schemaLocation

Answer

cloudFiles.includeExistingFiles

Question 5

A data engineer is using Databricks Auto Loader to ingest JSON files from an Azure Data Lake Storage Gen2 container into a Delta table. The engineer notices that the ingestion is slow and wants to optimize file discovery. The directory contains millions of files, and new files are added frequently. Which Auto Loader option should be used to improve file discovery performance?

Accepted Answer

Enable cloudFiles.useIncrementalListing.. Incremental listing leverages cloud storage APIs to list only new files, drastically reducing the time to discover files in large directories. Other options affect processing or schema handling but do not optimize file discovery, which is the bottleneck in this scenario.

Answer

Set cloudFiles.format to 'json'.

Answer

Set cloudFiles.maxFilesPerTrigger to a high value.

Answer

Enable cloudFiles.schemaEvolutionMode to 'rescue'.

Question 6

A data engineer is using Databricks Asset Bundles to deploy a data pipeline that includes a job and a notebook. The engineer wants to ensure that the deployment is idempotent and can be rolled back if needed. Which TWO of the following statements accurately describe the benefits of using Databricks Asset Bundles for this scenario? (Choose two.)

Accepted Answer

Databricks Asset Bundles allow you to define the entire pipeline as code, enabling version control and reproducible deployments.. Databricks Asset Bundles treat infrastructure as code, enabling version control and reproducible deployments. They allow you to define jobs, notebooks, and other resources in YAML, which can be committed to Git. This makes deployments idempotent and supports rollback by redeploying a previous version. The other options describe features not provided by Asset Bundles, such as automatic cluster creation, data synchronization, or mandatory Unity Catalog usage.

Answer

Databricks Asset Bundles require the use of Unity Catalog for all deployments.

Answer

Databricks Asset Bundles automatically synchronize data between different environments during deployment.

Answer

Databricks Asset Bundles automatically create a new cluster for each deployment to ensure isolation.

Question 7

A data engineer working in a Databricks workspace needs to create a new notebook that will be shared with teammates in the same workspace. They want the notebook to be organized in a folder named 'team_project' and be visible to all workspace users. Which action should the data engineer take to accomplish this in the Databricks workspace?

Accepted Answer

Navigate to the Workspace folder, create a new folder named 'team_project', then create the notebook inside it and set its permissions to 'Can View' for all workspace users.. In Databricks, the Workspace folder is the shared area where notebooks and folders can be organized and accessed by workspace users. Creating a folder named 'team_project' there and placing the notebook inside, then setting permissions to 'Can View' for all workspace users, directly meets the requirement. This method uses native workspace features without introducing unnecessary complexity.

Answer

Create the notebook in the Workspace folder, then use the Databricks CLI to set the notebook's permission to 'Can Manage' for the 'users' group.

Answer

Create the notebook in the Workspace folder under their user directory, then move it to a shared folder and grant 'Can Edit' permission to all workspace users.

Answer

Use the Databricks Repos feature to create a new repository, add a notebook, and set the repository permissions to 'Can Read' for all workspace users.

Question 8

A data engineer is using Auto Loader to ingest JSON files from a cloud storage directory into a Delta table. The directory receives files continuously, and the engineer wants to ensure that the ingestion process can handle schema drift where new columns are added to the JSON files over time. The engineer also wants to minimize the need to reprocess all data when the schema changes. Which configuration should the engineer use?

Accepted Answer

Set the option 'cloudFiles.schemaEvolutionMode' to 'addNewColumns' and enable 'mergeSchema' on the write stream.. To handle schema drift with Auto Loader, the 'cloudFiles.schemaEvolutionMode' should be set to 'addNewColumns' to automatically add new columns to the schema. Additionally, the write stream must have 'mergeSchema' enabled to update the Delta table schema. The other modes either fail or rescue data without adding columns, and without 'mergeSchema' on write, the new columns won't be persisted.

Answer

Set the option 'cloudFiles.schemaEvolutionMode' to 'failOnNewColumns' and enable 'mergeSchema' on the write stream.

Answer

Set the option 'cloudFiles.schemaEvolutionMode' to 'addNewColumns' and set 'mergeSchema' to 'false' on the write stream.

Answer

Set the option 'cloudFiles.schemaEvolutionMode' to 'rescue' and enable 'mergeSchema' on the write stream.

Question 9

A data engineer is setting up a Databricks job that runs a notebook on a schedule. The job must process data from a source that is updated daily and write results to a Delta table. The engineer wants to ensure that if the job fails, it automatically retries up to three times. Which feature should the engineer configure in the job settings to achieve this?

Accepted Answer

In the job's task settings, set the maximum number of retries to 3 and configure the retry interval.. Databricks jobs provide a retry configuration at the task level. By setting the maximum number of retries to 3, the task will be automatically retried if it fails, up to three times. This can be configured in the job's task settings, along with a retry interval if needed. This feature is designed to handle transient failures without manual intervention.

Answer

Enable the 'Retry on failure' option in the notebook itself and set the retry count to 3.

Answer

Set the maximum concurrent runs to 1 and enable retries with a maximum of 3 retries.

Answer

Configure the job to use a job cluster with auto-restart enabled and set the number of retries to 3.

Question 10

A team is using the Databricks CLI in their CI/CD pipeline to deploy jobs and notebooks. They need the pipeline to authenticate to a production workspace without embedding a personal user's credentials. Which authentication method should they configure for the CLI?

Accepted Answer

A service principal with an OAuth token or client credentials configured in the CLI profile.. For CI/CD automation, Databricks recommends authenticating the CLI with a service principal rather than a personal user identity. Service principals are non-human accounts that can be granted scoped permissions, and the CLI supports OAuth client credentials or service principal tokens. This avoids coupling deployments to an employee's account and supports credential rotation, auditing, and least privilege in production pipelines.

Answer

A personal access token generated from a developer's Databricks account.

Answer

The workspace's built-in admin username and password stored in the CI environment variables.

Answer

An SSH key pair registered in the Databricks workspace under the deploying user's profile.

Question 11

A team uses Databricks Asset Bundles to define a job that must exist in both a staging and a production workspace with different cluster sizes. They want a single bundle definition that deploys correctly to both targets. Which configuration should they use?

Accepted Answer

Use a single bundle with `targets` for staging and production, and override cluster settings per target using target-specific variables or overrides in `databricks.yml`.. Databricks Asset Bundles are designed for multi-environment deployment. A single `databricks.yml` with `targets` for staging and production, combined with target-specific overrides for cluster configuration, lets one bundle definition produce environment-appropriate resources without duplicating the job definition.

Answer

Define two separate bundles, one per workspace, and deploy each with its own `databricks bundle deploy` command.

Answer

Hard-code the production cluster size in the bundle and use a cluster policy in staging to downsize it at runtime.

Answer

Deploy the bundle only to production and point the staging workspace at the production job through a shared workspace link.

Question 12

A data engineer needs to share a Delta table with an external partner who does not have a Databricks account. The engineer wants to provide read-only access to the table for a limited time, ensuring the partner cannot access any other data. Which Databricks feature should the engineer use?

Accepted Answer

Unity Catalog delta sharing with a recipient and expiration.. Delta Sharing is Databricks' open protocol for secure data sharing with external parties, including those without Databricks accounts. With Unity Catalog, you can create a share containing specific tables and grant access to a recipient. The recipient can be given an expiration date, and they can access the data using any Delta Sharing client. This provides granular, read-only access limited to the shared data.

Answer

Generate a personal access token (PAT) for the partner to query the table via JDBC.

Answer

Export the table to a CSV file and share it via a secure file transfer.

Answer

Create a Databricks SQL endpoint and provide the partner with a SQL query URL.

Question 13

A CI/CD pipeline runs unit tests against transformation logic before deploying to production. The tests must run on a Databricks cluster and produce a pass/fail result that fails the pipeline when assertions do not hold. Which implementation best fits this requirement?

Accepted Answer

Create a Databricks Job that runs a notebook containing assertions, and have the CI/CD pipeline trigger the job and check its run result.. Tests must run where the production code runs. Triggering a Databricks Job that executes assertion notebooks lets the CI/CD pipeline inspect the run result and fail on assertion errors. This validates behavior on the actual Databricks runtime rather than in a disconnected local environment.

Answer

Run pytest locally in the CI runner against a sample of production data downloaded from cloud storage.

Answer

Use the Databricks SQL Statement Execution API to run `SELECT` statements that compare row counts between source and target tables.

Answer

Configure the cluster's init script to run the test suite automatically whenever the cluster starts.

Question 14

A data engineering team is setting up a CI/CD pipeline for Databricks notebooks using Databricks Repos and a Git provider. They want to ensure that changes are tested before being merged and that production deployments are controlled. Which TWO practices should they implement? (Choose two.)

Accepted Answer

Run automated tests in a CI job on a feature branch using a dedicated test workspace before merging to main.. A robust CI/CD process for Databricks Repos runs automated tests on feature branches in a dedicated test workspace, then gates merges on CI success. Production deployment should be a separate, controlled step that updates the production Repo to a tagged release, providing an immutable and rollback-friendly reference. These two practices together enforce quality gates and controlled promotion, which are core to CI/CD with Databricks Repos.

Answer

Store production credentials in the Repo's notebooks as widgets so the CI job can read them during tests.

Answer

Allow developers to commit directly to the main branch in the production workspace to reduce merge conflicts.

Answer

Configure the production Repo to track the main branch and enable automatic pull on every commit.

Question 15

A data engineer maintains a Delta table named inventory.products with columns product_id, category, price, and updated_at. The engineer needs to create a new table that contains one row per category with the average price and the most recently updated product_id in that category. The query must be efficient and use only standard Databricks SQL. Which statement should the engineer run?

Accepted Answer

SELECT category, AVG(price) AS avg_price, MAX_BY(product_id, updated_at) AS latest_product FROM inventory.products GROUP BY category. MAX_BY is designed exactly for the pattern of retrieving a value associated with the maximum of another column within a group. Grouping by category and applying AVG for the price and MAX_BY for the product_id returns the average price and latest product per category in one aggregation. The alternatives rely on non-deterministic ordering or array indexing, which do not guarantee the row with the greatest updated_at.

Answer

SELECT category, AVG(price) AS avg_price, COLLECT_LIST(product_id)[0] AS latest_product FROM inventory.products GROUP BY category

Answer

SELECT category, AVG(price) AS avg_price, LAST(product_id) AS latest_product FROM inventory.products GROUP BY category ORDER BY updated_at

Answer

SELECT category, AVG(price) AS avg_price, FIRST(product_id) AS latest_product FROM inventory.products GROUP BY category ORDER BY updated_at DESC