Databricks-DE-Pro Developing Code (Python/SQL) — All Questions With Answers

Question 1mediummultiple choice

Study the full Python automation breakdown →

A data engineer is implementing a Delta Lake table with streaming ingestion. To ensure high-concurrency writes while maintaining data integrity, what configuration must be enabled for the table to avoid 'Optimistic Concurrency Control' conflicts?

Question 2hardmultiple choice

Study the full Python automation breakdown →

Which TWO of the following statements accurately describe the behavior of the 'MERGE' operation in Delta Lake regarding schema evolution and concurrency?

Question 3mediummultiple choice

Study the full Python automation breakdown →

Refer to the exhibit. A data engineer encounters this error while running a SQL query on a Delta table. Based on the error message, what is the most likely cause?

Exhibit

Error: AnalysisException: [UNRESOLVED_COLUMN.WITH_SUGGESTION] A column or function parameter with name 'user_id' cannot be resolved. Did you mean one of the following? ['userId', 'user_id_raw']

Question 4mediummultiple choice

Study the full Python automation breakdown →

You are performing a 'Vacuum' operation on a large Delta table to remove expired files. You notice that the operation is running for an extended period and consuming significant cluster resources. What is the most likely reason for this performance overhead?

Question 5hardmultiple choice

Study the full Python automation breakdown →

You have a large Delta table and you need to perform an update on a single row based on a complex condition. You have noticed that 'UPDATE' statements are slow. What is the most effective way to optimize this operation in a production pipeline?

Question 6mediummultiple choice

Study the full Python automation breakdown →

A Data Engineer is developing a Delta Live Tables (DLT) pipeline. They need to ensure that rows containing null values in the 'customer_id' column are dropped during the ingestion process. Which constraint syntax is correct?

Question 7hardmulti select

Study the full Python automation breakdown →

A Data Engineer is optimizing a PySpark application. They notice significant data skew on the 'region_id' column during a join operation. Which TWO techniques can the engineer use to mitigate this skew?

Question 8hardmultiple choice

Study the full Python automation breakdown →

An engineer wants to capture data lineage and metrics for a custom Python transformation function that processes a Spark DataFrame outside of DLT. What is the most effective way to track this?

Question 9mediummultiple choice

Study the full Python automation breakdown →

When using Auto Loader to ingest files from cloud storage, what is the primary purpose of the 'cloudFiles.schemaLocation' option?

Question 10hardmultiple choice

Study the full Python automation breakdown →

A Data Engineer needs to perform a full refresh of a Delta table while maintaining the table's existing permissions and history. Which approach is best?

Question 11mediummultiple choice

Study the full Python automation breakdown →

An engineer needs to ensure that a Delta table is only updated if a record with the same ID does not exist in the target table. Which MERGE clause should be used?

Question 12mediummultiple choice

Study the full Python automation breakdown →

Refer to the exhibit. An engineer is trying to run a VACUUM command on a Delta table but receives an error stating the retention threshold is too low. What is the implication of setting the configuration shown in the exhibit?

Exhibit

{
  "cluster_id": "0521-123456-abcde789",
  "spark_version": "13.3.x-scala2.12",
  "spark_conf": {
    "spark.databricks.delta.retentionDurationCheck.enabled": "false"
  }
}

Question 13mediummultiple choice

Study the full Python automation breakdown →

You have a Delta table that is frequently queried by ID. To optimize performance, you decide to implement Z-Ordering. Which column is the best candidate for Z-Ordering?

Question 14mediummultiple choice

Study the full Python automation breakdown →

A data engineer is writing a PySpark job that reads from a Delta table, applies several transformations, and writes the result to another Delta table. The engineer notices that the job occasionally fails with a FileNotFoundException during the write phase. The source table is frequently updated by another job. Which action should the engineer take to resolve this issue?

Question 15easymultiple choice

Study the full Python automation breakdown →

A data engineer is writing a PySpark script in a Databricks notebook to process a DataFrame. They need to cache the DataFrame to avoid recomputation across multiple actions. Which method should they use to cache the DataFrame in memory and on disk?

Question 16hardmultiple choice

Study the full Python automation breakdown →

A data engineer is working with a Delta table that has a column 'id' as the primary key. The table is updated frequently with new records, and the engineer needs to ensure that only the latest record for each 'id' is kept. The engineer decides to use MERGE to upsert data from a staging table. Which MERGE condition will correctly update existing records and insert new ones?

Question 17hardmultiple choice

Study the full Python automation breakdown →

A Data Engineer needs to implement a PySpark transformation that processes a large DataFrame partitioned by date. The transformation must apply a complex Python function to each row, but the function requires loading a large lookup table (about 2 GB) that should be reused across all executors. The engineer wants to minimize serialization overhead and avoid reloading the lookup table for every row. Which approach is most appropriate?

Question 18easymultiple choice

Study the full Python automation breakdown →

A data engineer needs to create a Delta table that enforces a constraint to ensure that the 'email' column contains a valid email format. The engineer wants to reject any rows that do not match the pattern. Which SQL statement should be used to add this constraint?

Question 19mediummultiple choice

Study the full Python automation breakdown →

A Data Engineer is developing a PySpark job that reads from a Delta table, performs several transformations, and writes the result to another Delta table. The engineer notices that the job is producing many small files in the target table, which degrades read performance. The engineer wants to reduce the number of files written without changing the partitioning of the target table. Which optimization technique should be applied?

Question 20mediummultiple choice

Study the full Python automation breakdown →

A data engineer is using Databricks Repos to manage a Python project. The project contains multiple notebooks and a Python module in a separate file. The engineer wants to import the module into a notebook. Which approach should be used to ensure the module is importable within the notebook?

Question 21hardmulti select

Study the full Python automation breakdown →

A data engineer is optimizing a PySpark job that processes a large Delta table. The job performs a groupBy on a high-cardinality column and then joins the result with another table. The engineer notices that the job is slow and suspects data skew. Which two actions should the engineer take to mitigate the skew? (Choose two.)

Question 22mediummulti select

Study the full Python automation breakdown →

A Data Engineer is writing a PySpark job that reads a large Parquet dataset, performs a join with a smaller dimension table, and writes the result to a Delta table. The engineer wants to optimize the job for performance and cost. Which TWO actions should the engineer take? (Choose two.)

Question 23hardmulti select

Study the full Python automation breakdown →

A data engineer is implementing a Delta Live Tables (DLT) pipeline that ingests data from a streaming source. The pipeline must ensure that data is processed exactly once and that the pipeline can recover from failures without data loss or duplication. Which two configurations or practices are required to achieve exactly-once processing in DLT? (Choose two.)

Question 24easymultiple choice

Study the full Python automation breakdown →

A data engineer is writing a PySpark script to read a JSON file from cloud storage. The file contains nested fields and the engineer wants to flatten the structure to access nested attributes easily. Which PySpark function should be used to extract a nested field from a DataFrame column?

Question 25mediummultiple choice

Study the full Python automation breakdown →

You are optimizing a PySpark job that reads from a Delta table. You notice skewed data distribution on the 'customer_id' column, causing Task-level stragglers. Which transformation should you apply to the DataFrame to mitigate this skew during a join operation?

Question 26mediummultiple choice

Study the full Python automation breakdown →

You are processing sensitive PII data in a Delta table. You need to ensure that specific columns containing PII are not readable by general data analysts while maintaining the ability to perform aggregate analysis on those rows. Which feature should you implement?

Question 27hardmultiple choice

Study the full Python automation breakdown →

A data engineer is tuning a Spark job and decides to use 'Z-Ordering' on a Delta table. Which THREE of the following are valid considerations when selecting columns for Z-Ordering?

Question 28hardmultiple choice

Study the full Python automation breakdown →

You are migrating a legacy ETL process to Delta Live Tables (DLT). You have an existing table defined with a complex transformation that involves a custom Python function using a third-party library. How should you structure this in DLT to ensure the function is available and correctly applied?

Question 29mediummultiple choice

Study the full Python automation breakdown →

When designing a streaming pipeline using Structured Streaming, which THREE of the following are necessary to ensure 'exactly-once' processing semantics in Databricks?

Question 30mediummultiple choice

Study the full Python automation breakdown →

You have a large Spark DataFrame that you need to filter and save as multiple smaller Parquet files based on the values in a 'region' column. Which method should you use to optimize the file layout for subsequent queries?

Question 31hardmultiple choice

Study the full Python automation breakdown →

Your team is using a shared cluster for development. A user reports that their job is slow because the cluster memory is frequently filled by large data broadcasts. What configuration adjustment should you make to prevent this issue across all jobs on the cluster?

Question 32mediummultiple choice

Study the full Python automation breakdown →

Refer to the exhibit. You are appending data to an existing Delta table. What is the most likely cause of this error, and how should you resolve it?

Exhibit

Error: Py4JJavaError: An error occurred while calling o68.save. : org.apache.spark.sql.AnalysisException: Cannot write incompatible data to table 'sales_data': Column 'price' (decimal(10,2)) cannot be cast to 'price' (decimal(8,2)).

Question 33hardmultiple choice

Study the full Python automation breakdown →

Which THREE of the following are benefits of using Delta Lake over standard Parquet files for your data lake storage?

Question 34easymultiple choice

Study the full Python automation breakdown →

Which command should be used to display the history of transactions performed on a Delta table, including operations like overwrites and updates?

Question 35mediummulti select

Study the full Python automation breakdown →

Which THREE of the following are benefits of using Delta Lake over standard Parquet files in Databricks?

Question 36mediummultiple choice

Study the full Python automation breakdown →

You are writing a Spark application that uses a Broadcast Hash Join. You want to force the join to use broadcast optimization for a specific table. How do you implement this in PySpark?

Question 37mediummultiple choice

Study the full Python automation breakdown →

Which of the following describes the behavior of the 'OPTIMIZE' command in Databricks?

Question 38easymultiple choice

Study the full Python automation breakdown →

When partitioning a large Delta table, what is the best practice regarding the number of unique values in the partition column?

Question 39hardmultiple choice

Study the full Python automation breakdown →

When running a PySpark job, you receive an 'Out of Memory (OOM)' error during a shuffle operation. Which configuration is the most appropriate to address this first?

Question 40mediummulti select

Study the full Python automation breakdown →

Which TWO of the following are valid ways to trigger a job in Databricks?

Question 41mediummultiple choice

Study the full Python automation breakdown →

A Data Engineer is developing a Delta Live Tables (DLT) pipeline using Python. They need to ensure that records failing a specific data quality check are dropped, but the pipeline continues to process the remaining valid records. Which expectation syntax should the engineer implement?

Question 42mediummulti select

Study the full Python automation breakdown →

Which TWO of the following are valid ways to pass configuration parameters to a Databricks Job task? (Select TWO)

Question 43hardmultiple choice

Study the full Python automation breakdown →

When using the Unity Catalog, how should an engineer properly reference a table named 'sales' located in the 'finance' schema within the 'prod_catalog' catalog using Spark SQL?

Question 44mediummulti select

Study the full Python automation breakdown →

Which THREE features are provided by Delta Lake when compared to standard Parquet files? (Select THREE)

Question 45mediummultiple choice

Study the full Python automation breakdown →

A Data Engineer needs to perform an 'upsert' operation on a Delta table. Which command provides the most efficient way to merge new data with existing records in a single transactional step?

Question 46mediummultiple choice

Study the full Python automation breakdown →

A Data Engineer is using Databricks Asset Bundles (DABs) to manage a project. Where should the engineer define the job settings, such as clusters, schedules, and task dependencies?

Question 47hardmultiple choice

Study the full Python automation breakdown →

Which property must be set to ensure a Spark Structured Streaming query can handle changes to the source data schema, such as adding a new column?

Question 48mediummultiple choice

Study the full Python automation breakdown →

An engineer is writing a Python function to process data in a Databricks Notebook. Which command should they use to ensure that secrets, such as API keys, are not hardcoded or exposed in the plain text of the notebook?

Question 49mediummulti select

Study the full Python automation breakdown →

Which TWO of the following are benefits of using the Databricks Delta Lake 'Optimize' command? (Select TWO)

Question 50mediummultiple choice

Study the full Python automation breakdown →

You are writing a PySpark script to join two large tables. You want to ensure the join operation is optimized for performance by broadcasting the smaller table. Which configuration property should you adjust, or code construct should you use, to force this behavior?

Question 51hardmultiple choice

Study the full Python automation breakdown →

You are developing a Delta Live Tables (DLT) pipeline and need to ensure high data quality. Which TWO of the following statements correctly describe how Expectations work within DLT?

Question 52mediummultiple choice

Study the full Python automation breakdown →

You are using the Databricks CLI to automate workspace tasks. Which THREE of the following statements correctly identify capabilities of the Databricks CLI?

Question 53hardmultiple choice

Study the full Python automation breakdown →

You are debugging a PySpark job that is experiencing severe memory pressure during a join on a massive column. You suspect data skew. Which approach is best to mitigate this issue?

Question 54mediummultiple choice

Study the full Python automation breakdown →

Which of the following describes the correct use of a UDF (User Defined Function) in PySpark for production pipelines?

Question 55mediummultiple choice

Study the full Python automation breakdown →

When designing a production-grade data pipeline in Databricks, what is the recommended approach for managing secrets such as database credentials?

Question 56mediummultiple choice

Study the full Python automation breakdown →

A data engineer is building a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate late-arriving data up to 10 minutes and update aggregations accordingly. The engineer wants to use a watermark on the event-time column. Which code snippet correctly applies the watermark and performs a 5-minute tumbling window aggregation?

Question 57mediummultiple choice

Study the full Python automation breakdown →

A data engineer is developing a PySpark job that reads a large Delta table, performs a groupBy on a high-cardinality column, and writes the result to another Delta table. The job is experiencing performance issues due to data skew. The engineer wants to optimize the shuffle by using salting. Which approach correctly implements salting to distribute the skewed keys evenly?

Question 58hardmulti select

Study the full Python automation breakdown →

A data engineer is using Delta Live Tables (DLT) to build a pipeline that ingests data from a streaming source. The pipeline must ensure that the target table is updated incrementally and that data quality constraints are enforced. The engineer wants to use expectations to drop invalid records while maintaining pipeline performance. Which TWO of the following are true regarding DLT expectations and their behavior? (Choose two.)

Question 59mediummultiple choice

Study the full Python automation breakdown →

A data engineer is optimizing a PySpark job that processes a large DataFrame and writes the result to a Delta table. The job currently uses repartition(100) before writing, but the output consists of many small files. The engineer wants to reduce the number of output files without shuffling the entire dataset again. Which approach should be used?

Question 60mediummultiple choice

Study the full Python automation breakdown →

A Data Engineer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must handle late-arriving data up to 2 hours and produce correct aggregations per 10-minute window. The engineer wants the streaming query to automatically clean up old state so the job does not accumulate unbounded state. Which combination of Structured Streaming features should be used?

Question 61hardmultiple choice

Study the full Python automation breakdown →

A data engineer is implementing a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. The engineer needs to ensure that the job can recover from failures without data loss or duplication. The job uses foreachBatch to perform upserts into the Delta table. Which checkpointing configuration is required to achieve exactly-once semantics?

Question 62hardmultiple choice

Study the full Python automation breakdown →

A data engineer is using Delta Live Tables (DLT) to create a pipeline that processes streaming data. They need to ensure that the pipeline only processes new data since the last run and that the pipeline can recover from failures without reprocessing all data. Which combination of features should they use to achieve this?

Question 63easymultiple choice

Study the full Python automation breakdown →

A data engineer needs to read a CSV file from cloud storage into a Spark DataFrame in Databricks. The file has a header row and uses commas as delimiters. The engineer wants to infer the schema automatically. Which code snippet correctly reads the file?

Question 64mediummultiple choice

Study the full Python automation breakdown →

A data engineer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must handle late-arriving data and update already processed aggregates. Which watermark strategy should be used to allow updates to aggregates while bounding state store growth?

Question 65hardmultiple choice

Study the full Python automation breakdown →

A data engineer is using PySpark to process a large DataFrame and needs to reduce the number of partitions before writing to a Delta table to avoid creating too many small files. The DataFrame currently has 2000 partitions, each about 10 MB. The engineer wants to reduce the number of partitions to approximately 200 while minimizing data shuffling. Which approach is most appropriate?

Question 66mediummultiple choice

Study the full Python automation breakdown →

A data engineer is developing a PySpark job that reads from a Delta table and performs a series of transformations. The engineer notices that the job is slow and suspects that the query plan is not optimized because statistics are outdated. Which command should the engineer run to update the statistics for the Delta table to improve query performance?

Error: Py4JJavaError: An error occurred while calling o68.save. : org.apache.spark.sql.AnalysisException: Cannot write incompatible data to table 'sales_data': Column 'price' (decimal(10,2)) cannot be cast to 'price' (decimal(8,2)).