Databricks-Spark-Assoc Practice Test 6 — 10 Questions

Question 1

Which TWO of the following are benefits of using Delta Lake over standard Parquet files in Spark SQL?

Accepted Answer

Support for ACID transactions.. Delta Lake extends the capabilities of standard Parquet by adding a transaction log and metadata layer. This enables ACID transactions, Time Travel, and schema enforcement, which are critical for robust data engineering. For a Databricks certified developer, knowing why Delta is the preferred format for the Lakehouse architecture is essential for building scalable, reliable, and maintainable data systems that surpass the limitations of raw file-based storage.

Answer

Faster raw I/O performance for single-column reads.

Answer

Automatic conversion of CSV files to optimized Parquet.

Answer

Compatibility with legacy Hive Metastore versions without modification.

Question 2

A developer is migrating a local pandas script to the Pandas API on Spark. The dataset is large and partitioned across many executors. The developer executes a custom row-wise operation using a standard Python lambda function inside a `.apply()` method without specifying return types or using vectorized operations. Why might this approach cause performance degradation in Databricks?

Accepted Answer

Iterating through rows via Python lambdas forces high data serialization overhead between JVM and Python workers, destroying vectorized execution benefits.. Standard pandas `.apply()` functions often execute row-by-row python processing rather than leveraging native Catalyst query optimizations. When using non-vectorized operations in Spark without explicit type hints, the engine must serialize data between JVM and Python workers repeatedly. This causes high serialization overhead, defeats distributed columnar optimization, and ultimately results in severe performance degradation compared to vectorized Spark expressions.

Answer

Spark automatically converts all pandas apply operations into native GPU-accelerated C++ code during the initial logical plan compilation phase.

Answer

The Catalyst optimizer completely bypasses the execution plan, forcing the cluster to fall back to a single-threaded local driver execution model.

Answer

Pandas API on Spark strictly prohibits the use of the `.apply()` method and immediately throws a compilation error during execution.

Question 3

You are running a Structured Streaming query on Databricks that reads from a Kafka topic and writes to a Delta table. The query uses `option("maxOffsetsPerTrigger", 10000)` to limit the number of records per micro-batch. During a peak, the Kafka topic accumulates a large backlog. You notice that the query is processing data but the backlog is not decreasing. What is the most likely cause?

Accepted Answer

The `maxOffsetsPerTrigger` value is lower than the rate at which new data is arriving.. The backlog is not decreasing because the query is limited to processing 10000 records per micro-batch, which is likely less than the incoming rate. This cap is set by `maxOffsetsPerTrigger`. To reduce the backlog, you must increase this limit or the processing capacity. The other options do not address the rate mismatch: the option is not ignored, triggers do not overlap, and starting offsets only affect initial reads.

Answer

The Kafka source is not configured with `startingOffsets` set to `latest`.

Answer

The trigger interval is too short, causing the query to start new micro-batches before the previous one finishes.

Answer

The `maxOffsetsPerTrigger` option is ignored when writing to Delta Lake.

Question 4

What is the function of the 'Executor' within the Spark execution model?

Accepted Answer

It executes tasks assigned by the Driver and caches data.. The executor is the component that performs the actual computation and stores data. It is important to know that executors are distributed entities—they carry out the work assigned by the Driver. By understanding that executors run tasks in parallel and manage the cache, developers can configure their cluster appropriately for the memory and compute needs of their specific data processing pipelines, ensuring high throughput and efficient resource utilization throughout the application's lifecycle.

Answer

It acts as the central coordinator for all worker nodes.

Answer

It initializes the SparkSession for the user's application.

Answer

It defines the logical plan of the Spark application.

Question 5

An analyst has a Spark SQL DataFrame named events with a string column event_time in the format 'yyyy-MM-dd HH:mm:ss'. They want to add a new column event_date containing only the date portion, keeping the original column intact. Which expression should they use in a select statement?

Accepted Answer

to_date(event_time, 'yyyy-MM-dd HH:mm:ss'). Using to_date with the matching pattern correctly parses the timestamp string and returns a date-typed column containing only the date component. The other functions either return strings, operate on date types rather than strings, or rely on fragile character extraction, so they do not produce a proper date column as required.

Answer

date_format(event_time, 'yyyy-MM-dd')

Answer

trunc(event_time, 'MM')

Answer

substring(event_time, 1, 10)

Question 6

A developer is using Spark Connect to connect to a Databricks cluster. They attempt to read a CSV file from a path that exists on the client machine's local disk. The operation fails with a file not found error. What is the most likely reason for this failure?

Accepted Answer

The file path is interpreted relative to the server's file system, not the client's, so the server cannot find the file.. Spark Connect operates in a client-server model where the server executes all data operations. File paths are resolved on the server's file system. If a developer refers to a local file on the client machine, the server cannot access it, causing a file not found error. The correct approach is to place data in a location accessible to the cluster, such as cloud storage.

Answer

The CSV reader requires a schema to be explicitly provided when using Spark Connect.

Answer

Spark Connect does not support reading CSV files; only Parquet and Delta are supported.

Answer

The client must first upload the CSV file to the server's local file system using spark.uploadFile().

Question 7

A data analyst needs to run a Spark SQL query that returns the top 5 highest-paid employees from a table named employees, ordered by salary descending. Which query correctly returns exactly five rows?

Accepted Answer

SELECT * FROM employees ORDER BY salary DESC LIMIT 5. To get the top 5 highest-paid employees, the query must sort by salary descending and then apply LIMIT 5. Spark SQL supports ORDER BY followed by LIMIT, which returns the first five rows after sorting, exactly matching the requirement.

Answer

SELECT TOP 5 * FROM employees ORDER BY salary DESC

Answer

SELECT * FROM employees ORDER BY salary DESC FETCH FIRST 5 ROWS ONLY

Answer

SELECT * FROM employees LIMIT 5 ORDER BY salary DESC

Question 8

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Accepted Answer

Broadcast join for small tables.. Reducing network traffic is critical in Spark performance tuning. By utilizing features like broadcast joins, you avoid the shuffle phase entirely when one table is small. Additionally, using filter and select operations early in the transformation pipeline ensures that only the necessary rows and columns are transmitted, significantly decreasing the total data volume shuffled during wide transformations.

Answer

Increase spark.sql.shuffle.partitions.

Answer

Enable dynamic resource allocation.

Answer

Increase spark.driver.memory.

Question 9

A streaming job using 'mapGroupsWithState' is failing due to excessive memory usage. Which strategy is most effective for mitigating this?

Accepted Answer

Implementing state timeouts.. The 'mapGroupsWithState' function maintains state for keys until they are explicitly timed out or removed. If keys are not removed, the state grows indefinitely, leading to memory issues. Using 'GroupStateTimeout' (either processing or event-time) allows the engine to automatically expire state after a specific duration, effectively bounding memory usage. This is a critical pattern for managing state in long-running streaming applications on Databricks.

Answer

Increasing the number of partitions.

Answer

Reducing the trigger interval.

Answer

Switching to Append mode.

Question 10

You are debugging a PySpark DataFrame job on Databricks that performs multiple transformations and actions on a large delta table. You notice that the execution plan shows redundant computations where the same upstream DataFrame is evaluated repeatedly. Which transformation should you apply to optimize this workflow and avoid recomputing the upstream lineage?

Accepted Answer

Call df.cache() or df.persist() before referencing the DataFrame in multiple downstream actions.. Persisting or caching a DataFrame instructs Spark to store intermediate results in memory or disk across actions, preventing expensive recomputation of the entire lineage graph. This optimization is critical for iterative algorithms or workflows where a single DataFrame is referenced multiple times downstream.

Answer

Call df.broadcast() on the DataFrame before each downstream join operation.

Answer

Call df.repartition() to distribute the data evenly across partitions before execution.

Answer

Call df.coalesce() to reduce the number of shuffle partitions prior to the final write operation.