Databricks-DE-Assoc Practice Test 5 — 25 Questions

Question 1

Which TWO of the following statements are correct regarding the use of Databricks Repos for CI/CD?

Accepted Answer

Databricks Repos allows users to integrate with Git providers like GitHub, GitLab, and Bitbucket.. Databricks Repos facilitates seamless Git integration within the Databricks workspace. It enables teams to perform development directly in the workspace while keeping code versioned in remote Git repositories. By understanding that Repos maps workspace folders to Git branches, engineers can implement effective branching strategies and merge workflows. This integration bridges the gap between collaborative notebook development and standard software engineering practices, ensuring that changes are tracked, reviewed, and tested before deployment to production environments.

Answer

Databricks Repos can only be used for Python code and does not support SQL or Markdown files.

Answer

Each Databricks Repo is a local-only folder that cannot be synchronized with a remote Git repository.

Answer

Changes made in a Repo automatically deploy to production without needing to be merged in Git.

Question 2

A junior data engineer notices that a scheduled Databricks job running a heavy ETL notebook is failing intermittently due to cluster driver out-of-memory errors. Which TWO configuration changes or architectural adjustments should be implemented to resolve this issue? (Select exactly TWO)

Accepted Answer

Select a larger driver node instance type with more memory capacity.. Driver out-of-memory errors typically occur when the driver node collects too much data into local memory using actions like collect() or handles excessive broadcast joins. Upgrading to a driver node with more RAM provides immediate headroom, while refactoring code to avoid pulling massive datasets to the driver prevents memory exhaustion fundamentally.

Answer

Increase the maximum number of worker nodes in the autoscaling cluster configuration.

Answer

Enable Delta caching on the cluster workers to offload data from the driver.

Answer

Decrease the Spark SQL shuffle partitions default setting to a lower number.

Question 3

A data engineering team needs to restrict access to a sensitive customer table in Unity Catalog so that junior data engineers can only view non-PII columns, while senior engineers can view all columns. Which approach should be used to implement this requirement securely and efficiently?

Accepted Answer

Define a SQL user-defined function combined with a dynamic view utilizing the IS_MEMBER function to conditionally mask sensitive columns based on the querying user's group membership.. Row and column-level security in Unity Catalog is implemented by defining SQL user-defined functions and applying dynamic views or masking policies. Creating a dynamic view using the IS_MEMBER function allows the database engine to evaluate user group membership at query execution time and conditionally mask or exclude sensitive columns without duplicating the underlying physical dataset, ensuring optimal storage and compliance.

Answer

Create separate physical copies of the customer table for junior and senior engineers, applying access control lists directly on the underlying files in cloud object storage using IAM policies.

Answer

Revoke ALL PRIVILEGES on the customer table from the junior engineers group and grant SELECT privilege only after they sign a data access request form stored in a shared workspace notebook.

Answer

Apply table ACLs directly on the raw Delta table by executing GRANT SELECT ON TABLE customer TO `junior-engineers` while writing custom Spark UDFs in every notebook to drop PII columns.

Question 4

A data engineer needs to configure a Databricks Job containing three distinct tasks: ingest, transform, and report. The transform task must only execute if the ingest task completes successfully, but the report task should execute regardless of whether the transform task succeeds or fails. How should the task dependencies be configured?

Accepted Answer

Configure transform to depend on ingest, and do not include transform as a dependency for report.. Task dependencies in Databricks Jobs are managed by defining upstream parent tasks using the depends_on field. To make transform run after ingest, transform depends on ingest. To make report run regardless of transform's outcome, report must not depend on transform, or should be configured independently. Correctly structuring task dependencies ensures downstream consumers do not ingest corrupt data while maintaining resilience for reporting tasks.

Answer

Configure transform to depend on ingest, and configure report to depend on transform.

Answer

Configure ingest to depend on transform, and configure report to depend on ingest.

Answer

Configure both transform and report to depend on ingest simultaneously.

Question 5

Refer to the exhibit. When is this job scheduled to run?

Accepted Answer

At 12:00 PM every day.. Understanding Quartz cron syntax is fundamental for scheduling jobs in Databricks. The expression '0 0 12 * * ?' maps to 12:00 PM (noon) daily. Being able to interpret and define schedules correctly is vital for data engineers to ensure that data pipelines run at the exact times required by business stakeholders, ensuring that data is fresh and available for reporting during peak business hours without unnecessary delays.

Answer

Every minute at 12 seconds.

Answer

Every 12 hours.

Answer

Every Monday at 12:00 AM.

Question 6

A data engineer is designing a Delta Lake table that will be used for both batch and streaming reads. The table must support upserts from a streaming source and maintain ACID transactions. The engineer wants to ensure that the table can be efficiently queried by downstream consumers using SQL while minimizing storage costs. Which TWO actions should the engineer take to meet these requirements? (Choose two.)

Accepted Answer

Use the MERGE INTO command to perform upserts from the streaming source into the Delta table.. To support streaming upserts with ACID transactions and efficient SQL queries while minimizing storage costs, the engineer should use MERGE INTO for upserts and regularly run OPTIMIZE with Z-ORDER. MERGE INTO ensures atomic upserts, and OPTIMIZE with Z-ORDER compacts files and improves data skipping, reducing storage and query costs. Other options either add overhead, degrade performance, or remove essential Delta Lake features.

Answer

Enable change data feed on the Delta table to allow downstream consumers to read change events.

Answer

Partition the Delta table by a high-cardinality column to improve query performance.

Answer

Convert the Delta table to a Parquet table to reduce storage costs.

Question 7

A data engineer is working with a Delta table that contains a column named raw_data of type STRING, which holds JSON strings. The engineer needs to extract specific fields from this JSON and store them as separate columns in a new Delta table. Which approach is most efficient and maintains data quality?

Accepted Answer

Use the from_json function with a predefined schema to parse the JSON strings, then select the desired fields and write them to a new Delta table.. The most efficient and quality-preserving method is to use from_json with a predefined schema. This function parses JSON strings into a structured format according to the specified schema, enabling direct extraction of fields with correct data types. It also provides error handling options, such as permissive or failfast modes, to manage malformed JSON. This approach minimizes data shuffling and ensures that the resulting table has consistent, validated data, which is essential for downstream analytics.

Answer

Use the json_tuple function to extract multiple fields in one pass, specifying the field names as arguments, and then write the results to a new Delta table.

Answer

Use the get_json_object function for each field to extract values from the raw_data column, then create a new DataFrame with these columns.

Answer

Use the schema_of_json function to infer the schema from a sample of the raw_data, then apply from_json with the inferred schema to parse the JSON.

Question 8

A data engineer runs a Structured Streaming job that writes to a Delta table. The job processes data from a Kafka topic and uses a 10-minute watermark. After a few hours, the engineer notices that the streaming query's input rate is steady, but the processing rate has dropped significantly, and the batch duration has increased from 5 seconds to over 2 minutes. The job is running on a cluster with autoscaling enabled. Which action should the engineer take FIRST to diagnose the performance degradation?

Accepted Answer

Examine the Streaming Query progress metrics in the Spark UI, focusing on the 'addBatch' and 'walCommit' durations.. The Streaming Query progress metrics in the Spark UI are the primary tool for diagnosing Structured Streaming performance. They break down batch time into components like addBatch and walCommit, showing whether the delay is in data processing or Delta Lake commit operations. This targeted diagnosis avoids premature tuning and guides the engineer to the actual bottleneck.

Answer

Increase the number of shuffle partitions by setting spark.sql.shuffle.partitions to a higher value.

Answer

Restart the streaming job with a larger driver node to handle the increased metadata load.

Answer

Repartition the Kafka source topic to increase parallelism in the streaming query.

Question 9

A data engineer maintains a Lakeflow Job with a scheduled trigger set to run every day at 08:00. The job's source table is refreshed by an upstream process that sometimes finishes later than expected, causing the job to process stale data. The engineer wants the job to start only after the upstream refresh completes, regardless of the clock time, while still preserving the existing 08:00 schedule as a fallback. Which trigger configuration should the engineer implement?

Accepted Answer

Add a table update trigger on the upstream table while keeping the scheduled trigger, so the job runs when the table refreshes or at 08:00.. The requirement is a job that starts when upstream data is ready, yet still runs at 08:00 if that event has not occurred. A table update trigger detects upstream commits and starts the job promptly, while the retained scheduled trigger supplies the fallback. Together they deliver event-driven orchestration with a time-based safety net, which neither a pure schedule nor a polling file trigger provides.

Answer

Convert the scheduled trigger to a continuous trigger so the job restarts immediately after each run.

Answer

Add a file arrival trigger that monitors the upstream table's storage location and remove the scheduled trigger.

Answer

Add a run-duration timeout to the scheduled trigger so the job fails if the upstream has not finished by 08:00.

Question 10

Refer to the exhibit. A data engineer is configuring an Auto Loader stream with the provided options. What will happen if a new JSON file arrives containing a field that is not currently in the target table schema?

Accepted Answer

The new field will be added to the target table automatically.. The schemaEvolutionMode configuration determines how Auto Loader reacts to changes in the source data structure. By setting this to addNewColumns, the engineer ensures that the pipeline remains operational when new fields appear. This is a key part of the Medallion architecture where bronze tables must capture all incoming raw data without strict validation rules.

Answer

The stream will fail and require a manual schema update.

Answer

The new field will be dropped and only existing columns are kept.

Answer

The data for the new field will be moved to a _rescued_data column.

Question 11

A data engineer is troubleshooting a Databricks job that intermittently fails with a `SparkException: Job aborted due to stage failure: Task not serializable`. The job reads from a Parquet file, performs a transformation using a custom function defined in a Python class, and writes to a Delta table. The engineer suspects that the custom function is causing the issue. Which action should the engineer take to resolve the serialization error?

Accepted Answer

Refactor the custom function to avoid referencing any non-serializable objects from the enclosing class or module.. The `Task not serializable` error arises when Spark tries to serialize a closure for execution on executors but encounters objects that cannot be serialized. This often happens when a function references a non-serializable object from its enclosing scope, such as a database connection or a large data structure. Refactoring the function to avoid such references ensures that only serializable data is captured, resolving the error.

Answer

Mark the custom function with the `@staticmethod` decorator to avoid serializing the enclosing class.

Answer

Increase the driver memory to ensure that the serialized function fits in memory.

Answer

Set the Spark configuration `spark.serializer` to `org.apache.spark.serializer.KryoSerializer` to enable Kryo serialization.

Question 12

A data engineer is designing a Databricks Job workflow. Which TWO of the following are valid ways to trigger a Databricks Job?

Accepted Answer

Using the 'Run now' button in the UI.. Databricks Jobs offer multiple entry points for execution. Understanding these triggers is essential for integrating pipelines into broader CI/CD cycles or event-driven architectures. By mastering these methods, engineers can effectively decouple their data processing logic from the scheduling mechanism, allowing for both manual verification during development and robust automated execution in production environments through APIs or service principals.

Answer

Directly editing the Python source code file.

Answer

Changing the cluster node type.

Answer

Modifying the job owner permissions.

Question 13

You are building a pipeline where a Bronze table contains JSON data with a nested 'user_info' struct. You need to promote this to a Silver table where 'user_id' is a top-level column. Which approach is the most efficient for this transformation?

Accepted Answer

Use the select() transformation with dot notation, e.g., 'user_info.user_id'.. Using Spark's native column selection and dot notation is the most efficient way to flatten nested structures. This avoids complex UDFs or manual parsing, which are slow and difficult to maintain. By flattening data early in the pipeline, downstream consumers can easily access specific fields without repeatedly parsing the nested JSON structure, which enhances the usability of the Silver-layer data.

Answer

Write a custom Python UDF to parse the JSON string and extract the field.

Answer

Cast the entire column to a String type and use regex to extract the ID.

Answer

Convert the DataFrame to an RDD to iterate over rows and manually extract values.

Question 14

Which THREE of the following represent core security principles enforced by Unity Catalog?

Accepted Answer

Centralized access control. Unity Catalog simplifies security by moving away from cloud-specific credential management to a unified identity-based model. By centralizing access control, audit logs, and data lineage, it ensures consistent security posture across all Databricks workspaces. This consolidation reduces the complexity of managing disparate security policies, ensuring that governance is applied uniformly and that administrators can easily monitor and report on data access across the entire data estate.

Answer

Direct cloud storage access via IAM

Answer

Workspace-specific metastores

Question 15

A data engineer wants to ensure that a Databricks Job task only runs if the preceding task completes successfully, but needs to add a specific timeout threshold for this individual task. Where should this configuration be applied?

Accepted Answer

Within the individual task definition.. In Databricks Jobs, task-level configurations such as timeouts, retries, and dependencies are defined within the task definition itself. By specifying a timeout in the task settings, the engineer prevents runaway processes from consuming cluster resources indefinitely. Understanding this granularity is vital for cost management and workflow reliability, as it allows engineers to set different SLAs for ingestion versus transformation tasks within a single orchestrated pipeline.

Answer

At the job level settings.

Answer

Inside the cluster configuration JSON.

Answer

Using a global Workspace policy.

Question 16

A data engineer is designing a Delta Lake pipeline that processes streaming sales transactions. The schema evolves frequently, and the pipeline must handle these changes without manual intervention. Which feature should the engineer enable to support automatic schema updates while preventing data corruption?

Accepted Answer

Enable 'mergeSchema' in the DataFrameWriter options.. Schema evolution is critical in production pipelines to accommodate changing source data structures. Delta Lake handles this automatically when .option('mergeSchema', 'true') is used during write operations. This ensures that new columns are added to the metadata without requiring a table rebuild. Understanding this mechanism is vital for maintaining robust, automated pipelines in Databricks environments where source systems are prone to upstream schema modifications.

Answer

Set the table property 'delta.autoOptimize.optimizeWrite' to true.

Answer

Use the ALTER TABLE command to manually add missing columns before writing.

Answer

Set the 'overwriteSchema' option to true during every write.

Question 17

A data engineer needs to run a Databricks notebook that processes data stored in an external ADLS Gen2 location. The notebook must access the data securely without embedding credentials in the notebook code. The engineer has already configured a Unity Catalog external location with a storage credential. Which method should the engineer use to read the data?

Accepted Answer

Reference the external location path directly in the notebook using the Unity Catalog external location URL.. Unity Catalog external locations are designed to provide secure access to cloud storage using storage credentials managed by Unity Catalog. Once an external location is configured, data engineers can reference the path directly in their notebooks, and Unity Catalog handles authentication and authorization. This approach avoids embedding credentials in code, aligns with security best practices, and ensures that access is governed by Unity Catalog permissions.

Answer

Use a personal access token (PAT) stored in Databricks secrets to authenticate to ADLS Gen2.

Answer

Set Spark configuration spark.hadoop.fs.azure.account.oauth2.client.id and related properties in the notebook to authenticate.

Answer

Mount the ADLS Gen2 container to DBFS using a service principal and then read the data from the mount point.

Question 18

A data engineer has configured a Databricks Job with multiple dependent tasks forming a linear pipeline. Task A extracts data, Task B transforms it, and Task C loads it into a gold table. The pipeline runs daily. The team notices that if Task B fails due to an intermittent schema validation issue, the entire job run fails, but they want Task C to execute conditionally only if Task B succeeds, while alerting the on-call engineer immediately upon any failure. How should the task dependencies and conditional execution be configured?

Accepted Answer

Ensure Task C has a dependency pointing exclusively to Task B, and configure a job-level email notification for failures.. Configuring explicit task dependencies via the UI or API ensures that downstream tasks only execute upon the successful completion of their parents. By linking Task C strictly to Task B's success, you prevent corrupting downstream layers with partial or failed upstream transformations. This workflow pattern is essential for maintaining reliable data engineering pipelines in production Lakeflow environments.

Answer

Set Task C to run on failure of Task B to capture the exception state before alerting.

Answer

Convert Task B and Task C into a single notebook task and handle exceptions internally with try-except blocks.

Answer

Disable the task dependency between Task B and Task C and run them in parallel with different cluster configurations.

Question 19

When utilizing Databricks Asset Bundles, how should secrets (e.g., API keys, database credentials) be handled to ensure security during CI/CD?

Accepted Answer

Use the bundle's secret reference syntax to map variables to a secret scope.. Hardcoding secrets in configuration files is a critical security vulnerability. Databricks Asset Bundles support referencing secrets through the 'secrets' scope, which securely retrieves credentials at runtime. By decoupling sensitive data from the version-controlled YAML files, teams maintain security, avoid leaking credentials in Git, and simplify secret rotation. This practice is essential for enterprise security and ensures that sensitive infrastructure data is managed in alignment with the principle of least privilege.

Answer

Hardcode the secrets in the bundle's YAML configuration files.

Answer

Store secrets as environment variables in the Git repository.

Answer

Ask all developers to manually enter secrets in the production UI.

Question 20

A data engineer needs to configure a Databricks Job to orchestrate a data pipeline that includes a Python task, a SQL task, and a notebook task. The pipeline requires passing a dynamic run identifier from the Python task to the subsequent SQL and notebook tasks. Which mechanism should the data engineer use to achieve this task-to-task dependency parameter passing?

Accepted Answer

Use the dbutils.jobs.taskValues.set method in the Python task and reference it downstream using the task value syntax.. Databricks workflows natively support passing values between tasks using task values. A Python task can set a task value using dbutils.jobs.taskValues.set, which downstream tasks can then reference using standard task value interpolation syntax. This eliminates external storage dependencies, ensuring robust, serverless orchestration directly managed by the Databricks control plane.

Answer

Write the dynamic identifier to a shared Unity Catalog volume and read it inside the downstream tasks.

Answer

Store the identifier as an environment variable in the cluster configuration shared by all three tasks.

Answer

Modify the global job parameters dynamically via the Databricks REST API from within the running Python script.