Databricks-DE-Pro Developing Code (Python/SQL) Practice Test 4 — 15 Questions

Question 1

A Data Engineer is using Databricks Asset Bundles (DABs) to manage a project. Where should the engineer define the job settings, such as clusters, schedules, and task dependencies?

Accepted Answer

In a databricks.yml file within the project root.. Databricks Asset Bundles use YAML files, specifically those located in the bundles configuration folder, to manage infrastructure as code. Defining job settings in the `databricks.yml` file allows for version control and consistent deployments across different environments (dev, staging, prod). This approach ensures that the entire project state is replicable and managed through a standardized CI/CD pipeline, reducing manual configuration errors and increasing the speed of environment deployments.

Answer

In a hardcoded JSON file located in the /tmp/ directory.

Answer

Inside the Python code using the dbutils.job() library.

Answer

By manually clicking through the Databricks UI after deployment.

Question 2

Which property must be set to ensure a Spark Structured Streaming query can handle changes to the source data schema, such as adding a new column?

Accepted Answer

cloudFiles.schemaEvolutionMode = 'addNewColumns'. The `cloudFiles.schemaEvolutionMode` property allows the Auto Loader to adapt to schema changes automatically. In production streaming environments, this is vital because data sources often evolve over time. Without this, the streaming job would fail when it encounters a record that does not match the schema inferred at the start of the job, resulting in pipeline downtime and manual intervention to reset the state.

Answer

cloudFiles.inferSchema = 'false'

Answer

cloudFiles.maxFilesPerTrigger = 'auto'

Answer

cloudFiles.useIncrementalListing = 'true'

Question 3

An engineer is writing a Python function to process data in a Databricks Notebook. Which command should they use to ensure that secrets, such as API keys, are not hardcoded or exposed in the plain text of the notebook?

Accepted Answer

dbutils.secrets.get(scope='my_scope', key='my_key'). The dbutils.secrets.get() utility is the standard, secure way to retrieve sensitive information stored in Databricks Secret Scopes. By referencing the scope and key name, the secret value is fetched at runtime and remains masked in the notebook output, preventing accidental exposure of credentials. This is a mandatory practice in any secure data engineering environment to comply with security policies and prevent unauthorized access to downstream data sources.

Answer

os.environ.get('API_KEY')

Answer

open('/secret/path').read()

Answer

spark.conf.get('secret_key')

Question 4

Which TWO of the following are benefits of using the Databricks Delta Lake 'Optimize' command? (Select TWO)

Accepted Answer

It compacts small files into larger, more efficient Parquet files.. The OPTIMIZE command compacts small files into larger files, which significantly improves read performance by reducing the metadata overhead of listing thousands of small files. It also allows for Z-Ordering, which collocated related information in the same files, enabling data skipping during query execution. These optimizations are crucial for maintaining high-performance data lakes as datasets grow over time in production environments.

Answer

It automatically shrinks the data volume by dropping null values.

Answer

It converts Parquet files to CSV format for better compatibility.

Answer

It automatically updates the table statistics for the CBO.

Question 5

You are writing a PySpark script to join two large tables. You want to ensure the join operation is optimized for performance by broadcasting the smaller table. Which configuration property should you adjust, or code construct should you use, to force this behavior?

Accepted Answer

df.join(broadcast(small_df), 'id'). Broadcasting small tables is a critical optimization technique in Databricks to prevent expensive shuffles across the cluster. By utilizing the broadcast hint, developers explicitly instruct the Spark Catalyst optimizer to send the smaller table to all worker nodes. This minimizes data movement and significantly reduces latency during join operations. Understanding this mechanism is essential for building scalable ETL pipelines and ensuring efficient cluster resource utilization within the Databricks unified analytics platform.

Answer

spark.conf.set('spark.sql.shuffle.partitions', '1')

Answer

spark.sql('SET spark.sql.autoBroadcastJoinThreshold = -1')

Answer

df.repartition(100)

Question 6

You are developing a Delta Live Tables (DLT) pipeline and need to ensure high data quality. Which TWO of the following statements correctly describe how Expectations work within DLT?

Accepted Answer

You can use 'expect_or_fail' to stop the pipeline if a constraint is violated.. Expectations are a fundamental feature in Delta Live Tables that allow engineers to define data quality constraints directly within the DLT pipeline code. By specifying these rules, developers can monitor data health, quarantine corrupt rows, or fail pipelines when critical thresholds are breached. This declarative approach integrates governance into the data engineering workflow, ensuring that downstream consumers receive only validated, high-quality data while maintaining robust audit trails for compliance.

Answer

Expectations only support dropping rows that fail validation.

Answer

Expectations must be configured in the pipeline settings JSON file.

Answer

Expectations automatically fix invalid data values.

Question 7

You are using the Databricks CLI to automate workspace tasks. Which THREE of the following statements correctly identify capabilities of the Databricks CLI?

Accepted Answer

The CLI can manage and deploy Databricks Asset Bundles (DABs).. The Databricks CLI is a powerful tool for DevOps integration and workspace automation. It provides a programmatic interface for interacting with the Databricks control plane, enabling developers to manage notebooks, clusters, and jobs without relying on the GUI. Mastering the CLI is essential for implementing CI/CD pipelines, automating infrastructure deployment via Terraform, and managing large-scale workspace resources efficiently within a professional Databricks environment.

Answer

The CLI can be used to execute arbitrary SQL queries on a SQL Warehouse.

Answer

The CLI can directly modify the contents of a Delta table.

Question 8

You are debugging a PySpark job that is experiencing severe memory pressure during a join on a massive column. You suspect data skew. Which approach is best to mitigate this issue?

Accepted Answer

Implement a salted join by adding a random integer to the join key.. Data skew is a common performance bottleneck in distributed systems where one task takes significantly longer than others because it processes a disproportionate amount of data. By salting the skewed key—adding a random prefix or suffix—the data is distributed more evenly across the cluster partitions. This technique prevents executor memory overflow and allows for parallel processing of the skewed key, significantly improving overall job performance and reliability.

Answer

Increase the number of partitions using 'spark.sql.shuffle.partitions'.

Answer

Use the 'cache()' method on the skewed dataframe.

Answer

Enable AQE and increase the 'spark.sql.adaptive.skewJoin.enabled' setting.

Question 9

Which of the following describes the correct use of a UDF (User Defined Function) in PySpark for production pipelines?

Accepted Answer

Use Pandas UDFs to leverage vectorized operations and improve performance.. UDFs in PySpark move data from the JVM (Java Virtual Machine) to a Python process, which incurs significant serialization overhead. When possible, it is always best to use native Spark SQL functions, which operate directly on the JVM, providing better performance and native optimization by the Catalyst engine. If a UDF is unavoidable, using Pandas UDFs (Vectorized UDFs) with Apache Arrow is the recommended approach to minimize serialization costs.

Answer

Always write row-based Python UDFs for maximum flexibility.

Answer

UDFs automatically optimize data distribution across the cluster.

Answer

UDFs are the most efficient way to perform simple string manipulation.

Question 10

When designing a production-grade data pipeline in Databricks, what is the recommended approach for managing secrets such as database credentials?

Accepted Answer

Use the dbutils.secrets.get() method to retrieve credentials at runtime.. Hardcoding credentials in notebooks is a severe security risk. Databricks Secrets provides a centralized, secure way to store and manage sensitive information. By referencing secrets via the 'dbutils.secrets.get()' API, credentials are kept out of the source code and version control. This approach ensures that secrets can be rotated without modifying code and restricts access to those specifically authorized to view them through the Databricks access control policies.

Answer

Store credentials in a public GitHub repository.

Answer

Pass credentials as environment variables via cluster initialization scripts.

Answer

Encrypt credentials using a local Python library and save them to a file.

Question 11

A data engineer is building a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate late-arriving data up to 10 minutes and update aggregations accordingly. The engineer wants to use a watermark on the event-time column. Which code snippet correctly applies the watermark and performs a 5-minute tumbling window aggregation?

Accepted Answer

df.withWatermark("event_time", "10 minutes").groupBy(window("event_time", "5 minutes")).count(). The correct approach is to apply the watermark on the event-time column before the aggregation, with a delay that matches the late-data tolerance (10 minutes). The aggregation must use a window of the desired size (5 minutes). Placing the watermark after the aggregation or using mismatched durations leads to incorrect results or errors.

Answer

df.withWatermark("event_time", "10 minutes").groupBy("event_time").count()

Answer

df.withWatermark("event_time", "5 minutes").groupBy(window("event_time", "10 minutes")).count()

Answer

df.groupBy(window("event_time", "5 minutes")).count().withWatermark("event_time", "10 minutes")

Question 12

A data engineer is developing a PySpark job that reads a large Delta table, performs a groupBy on a high-cardinality column, and writes the result to another Delta table. The job is experiencing performance issues due to data skew. The engineer wants to optimize the shuffle by using salting. Which approach correctly implements salting to distribute the skewed keys evenly?

Accepted Answer

Add a random salt column to the DataFrame, group by the salted column, then remove the salt and re-aggregate.. Salting is a technique to distribute skewed keys by appending a random salt to the key before the shuffle, then grouping by the salted key, and finally removing the salt and re-aggregating to get the correct results. This spreads the load of a hot key across multiple partitions. The other options do not correctly implement salting or fail to address the skew.

Answer

Use repartition on the skewed column before the groupBy to increase parallelism.

Answer

Use broadcast join instead of groupBy to avoid the shuffle.

Answer

Set spark.sql.adaptive.enabled to true and rely on Adaptive Query Execution to handle skew automatically.

Question 13

A data engineer is using Delta Live Tables (DLT) to build a pipeline that ingests data from a streaming source. The pipeline must ensure that the target table is updated incrementally and that data quality constraints are enforced. The engineer wants to use expectations to drop invalid records while maintaining pipeline performance. Which TWO of the following are true regarding DLT expectations and their behavior? (Choose two.)

Accepted Answer

Expectations with the 'drop' action will remove records that violate the constraint from the target table.. DLT expectations support actions like warn, drop, and fail. The 'drop' action removes invalid records and continues, while 'fail' stops the pipeline. These actions apply to both streaming tables and materialized views, and are evaluated on every update, not just the initial load. Thus, the statements about 'drop' and 'fail' are correct.

Answer

When an expectation with 'drop' action is violated, the pipeline fails and must be restarted.

Answer

Expectations can only be applied to streaming tables, not to materialized views.

Answer

Expectations are evaluated only during the initial data load, not on subsequent incremental updates.

Question 14

A data engineer is optimizing a PySpark job that processes a large DataFrame and writes the result to a Delta table. The job currently uses repartition(100) before writing, but the output consists of many small files. The engineer wants to reduce the number of output files without shuffling the entire dataset again. Which approach should be used?

Accepted Answer

Use coalesce(10) before writing to reduce the number of partitions without a full shuffle.. coalesce is the correct choice because it reduces the number of partitions without a full shuffle, making it efficient for reducing output file count. repartition would cause an unnecessary shuffle, and changing shuffle partitions or using partitionBy does not achieve the goal of reducing files without a shuffle.

Answer

Use partitionBy("column") to write the data into subdirectories, which reduces the number of files per directory.

Answer

Set spark.sql.shuffle.partitions to 10 before writing.

Answer

Use repartition(10) before writing to redistribute data evenly and reduce file count.

Question 15

A Data Engineer is building a Structured Streaming pipeline that reads from a Kafka topic and writes to a Delta table. The pipeline must handle late-arriving data up to 2 hours and produce correct aggregations per 10-minute window. The engineer wants the streaming query to automatically clean up old state so the job does not accumulate unbounded state. Which combination of Structured Streaming features should be used?

Accepted Answer

Use a tumbling window of 10 minutes with a watermark of 2 hours on the event-time column, and set the output mode to append.. To handle late data up to 2 hours while producing 10-minute aggregations and bounded state, the pipeline needs event-time tumbling windows of 10 minutes and a watermark of 2 hours. Append mode emits a window result only after the watermark exceeds the window end, and Spark automatically evicts state older than the watermark, keeping state size proportional to the lateness bound rather than the entire stream history.

Answer

Use a sliding window of 10 minutes with a watermark of 10 minutes on the event-time column, and set the output mode to update.

Answer

Use a tumbling window of 10 minutes with a watermark of 2 hours on the processing-time column, and set the output mode to append.

Answer

Use a tumbling window of 2 hours with a watermark of 10 minutes on the event-time column, and set the output mode to complete.