Databricks-Spark-Assoc Practice Test 14 — 15 Questions

Question 1

A Databricks notebook job calls a PySpark UDF built on a Python function that performs string parsing. The job completes successfully on a small sample, but on the full production dataset it fails with a PythonException and the executor logs show high garbage collection time. Which change is the most appropriate first step to make the job reliable without changing business logic?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Replace the Python UDF with a Spark SQL built-in function such as regexp_extract or split, which runs inside the JVM and avoids Python serialization and per-row interpreter overhead.. A scalar Python UDF forces each row through a Python worker, incurring serialization, interpreter startup, and per-row overhead that shows up as GC pressure at scale. Rewriting the logic with Spark SQL built-in functions keeps execution in the JVM with whole-stage code generation, eliminating that overhead and typically resolving both the reliability and performance symptoms. It preserves the business logic while removing the root cause rather than masking it with more resources.

Answer

Set spark.sql.execution.arrow.pyspark.enabled to true so the UDF uses Arrow for every row it processes.

Answer

Mark the UDF with the deterministic flag and register it with spark.udf.register so Spark can cache its results automatically.

Answer

Increase spark.executor.memory and spark.executor.cores so each executor has more headroom to run the Python workers.

Question 2

A Spark application reads a large CSV file and performs a series of transformations, including a filter and a join with a small lookup table. The job is running slowly, and the Spark UI shows that the CSV parsing stage is taking a long time. Which action would most improve performance?

Accepted Answer

Convert the CSV to Parquet and read the Parquet file instead.. Converting the CSV to Parquet addresses the slow CSV parsing by leveraging Parquet's columnar storage, which enables column pruning and predicate pushdown. This reduces the amount of data read and parsed, directly improving the performance of the initial stage. The other options either do not target the parsing bottleneck or could worsen performance.

Answer

Increase the number of partitions when reading the CSV by setting a lower spark.sql.files.maxPartitionBytes.

Answer

Set spark.sql.autoBroadcastJoinThreshold to -1 to disable broadcast join.

Answer

Cache the CSV DataFrame immediately after reading.

Question 3

A PySpark DataFrame job on Databricks runs slowly. Inspection of the Spark UI shows that a shuffle stage writes 200 partitions but downstream stages process only a few, and the physical plan shows an Exchange before a filter. Which two changes are most likely to improve performance? (Choose two.)

Accepted Answer

Reduce the number of shuffle partitions or coalesce the output so fewer, larger partitions are produced.. The plan shows an Exchange feeding stages that process only a few partitions, meaning shuffle volume and partition sizing are both inefficient. Filtering earlier cuts the rows that ever reach the shuffle, and consolidating partitions removes the overhead of many tiny tasks. Together they reduce both the data moved across the network and the number of tasks scheduled, which is what the UI evidence points to. Resource or caching changes would not target either cause.

Answer

Increase spark.sql.shuffle.partitions to a much larger number so each task handles fewer rows.

Answer

Repartition the DataFrame on the join key before the shuffle to colocate matching rows.

Answer

Cache the shuffled DataFrame with persist so downstream stages reuse the same partitions.

Question 4

You are building a Structured Streaming job that reads from a Kafka source and writes to a Delta table. You need to ensure that the job can recover from failures and process data exactly once. Which TWO of the following are required to achieve exactly-once semantics? (Choose two.)

Accepted Answer

Use a checkpoint location to store progress information.. Exactly-once semantics in Structured Streaming require a reliable checkpoint location to track progress and an idempotent sink like Delta Lake that can handle replays without duplicating data. Checkpointing ensures the query can recover from failures, while Delta's transactional writes guarantee that each batch is applied exactly once.

Answer

Enable `spark.sql.streaming.forceDeleteTempCheckpointLocation` to clean up temporary files.

Answer

Configure the Kafka source with `startingOffsets` set to `earliest`.

Answer

Set the `maxOffsetsPerTrigger` option to limit the number of records per batch.

Question 5

A Databricks job joins a 500 GB sales table with a 300 GB returns table on a customer_id key. A few customer_id values account for a large fraction of rows on both sides, and the job fails with executor OOM during the join. The developer wants to distribute the hot keys across more partitions without changing the query logic. Which technique should be used?

Accepted Answer

Enable Adaptive Query Execution with skew join handling so Spark splits skewed partitions at runtime.. AQE skew join handling identifies partitions that are disproportionately large after the shuffle and splits them into smaller sub-partitions, each handled by a separate task. This spreads the hot customer_id values across executors and resolves the OOM. Raising shuffle partitions or repartitioning by the join key keeps each hot key in one partition, and broadcasting a 300 GB table is infeasible.

Answer

Broadcast the 300 GB returns table to every executor to avoid shuffling the skewed keys.

Answer

Increase spark.sql.shuffle.partitions to 8000 to spread the skewed keys across more tasks.

Answer

Repartition both DataFrames by customer_id with 8000 partitions before the join.

Question 6

A developer must combine two DataFrames, `left_df` and `right_df`, on a key column `id`. They need every row from `left_df` regardless of whether a match exists in `right_df`, and matching rows from `right_df` where available, with unmatched right-side columns filled as null. Which join invocation produces this result?

Accepted Answer

left_df.join(right_df, on="id", how="left_outer"). A left outer join retains every left row and fills right-side columns with null when no match exists, which is exactly what the scenario specifies. Inner join discards unmatched left rows, right outer join preserves the wrong side, and full outer join introduces unmatched right rows that were not requested.

Answer

left_df.join(right_df, on="id", how="right_outer")

Answer

left_df.join(right_df, on="id", how="inner")

Answer

left_df.join(right_df, on="id", how="full_outer")

Question 7

A developer is tuning a Spark job that performs a join between a large fact table and a medium-sized dimension table. The job suffers from data skew, with a few keys having a disproportionately large number of rows. The developer wants to mitigate the skew. Which two actions are most effective? (Choose two.)

Accepted Answer

Enable Adaptive Query Execution (AQE) and set spark.sql.adaptive.skewJoin.enabled to true.. Data skew during a join occurs when a few keys have many more rows than others, causing some tasks to run much longer. Adaptive Query Execution with skew join optimization dynamically splits skewed partitions, while manual salting distributes hot keys across multiple tasks. Both approaches balance the workload and reduce the impact of skew.

Answer

Broadcast the medium-sized dimension table to all executors.

Answer

Use a repartition on the join key before the join to evenly distribute the data.

Answer

Increase spark.sql.shuffle.partitions to a very high number.

Question 8

A developer is building a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. The job must handle late data up to 15 minutes and ensure that aggregations are updated correctly. The developer adds a watermark of 15 minutes on the event time column. What is the effect of this watermark on the aggregation state and output?

Accepted Answer

The watermark allows the engine to drop state for windows that are older than the watermark, and late data within 15 minutes is still processed and can update the aggregation.. A watermark in Structured Streaming defines a threshold for how late data can arrive. It allows the engine to drop old state once the watermark passes the window end time, preventing unbounded state growth. Late data within the watermark delay is still processed and can update aggregations. This balances correctness and resource usage, enabling efficient handling of late-arriving events in streaming aggregations.

Answer

The watermark ensures exactly-once processing by deduplicating records that arrive within the 15-minute window.

Answer

The watermark forces the output mode to complete, so the entire aggregation result is recomputed on every trigger.

Answer

The watermark causes the aggregation state to be dropped immediately after 15 minutes, and any late data beyond that is ignored.

Question 9

You are performing a stream-stream join between two streaming DataFrames, `orders` and `payments`, both with watermarks defined on their event time columns. The join condition is `orders.orderId == payments.orderId` and it is an inner join. What happens to state in this join?

Accepted Answer

State is maintained for both streams and is cleaned up based on the watermarks.. In a stream-stream inner join, the engine maintains state for both streams to match records across micro-batches. Watermarks define when state can be evicted: once the watermark passes the event time of a record plus the allowed lateness, its state is removed. This ensures that late-arriving matches within the watermark window are still processed.

Answer

State is maintained only for the right stream, as it is the stream being joined to.

Answer

State is not maintained; the join is stateless and relies on the current micro-batch.

Answer

State is maintained only for the left stream, as it is the driving stream.

Question 10

A streaming DataFrame `df` has a watermark defined on `eventTime` with a delay of 5 minutes. The query uses `withWatermark("eventTime", "5 minutes")` and writes to a Delta table in `append` output mode. A record with event time 10:00 arrives when the watermark is at 10:10. What happens to this record?

Accepted Answer

The record is dropped because its event time is older than the watermark.. With a 5-minute watermark, any record whose event time is more than 5 minutes behind the current watermark is considered late. In append mode, such records are dropped because the system assumes all data for that time has already been emitted. The record at 10:00 arrives when the watermark is 10:10, so it is 10 minutes late and is discarded. This matches the expected watermark behavior.

Answer

The record is held in state until the next watermark update and then emitted.

Answer

The record causes the query to fail with a watermark violation error.

Answer

The record is processed and appended to the Delta table immediately.

Question 11

You are developing a Structured Streaming job that reads from a Delta table and writes to another Delta table. You need to ensure that the streaming query can recover from failures and continue processing without data loss or duplication. Which of the following must be configured?

Accepted Answer

A checkpoint location. To recover from failures and ensure exactly-once processing, a Structured Streaming query must have a checkpoint location. This location stores the progress information and state, allowing the query to resume from where it left off. Watermarks, trigger intervals, and output modes are unrelated to basic fault tolerance. Therefore, the checkpoint location is the essential configuration.

Answer

An output mode of `complete`

Answer

A watermark on the event time column

Answer

A trigger interval of `processingTime='1 second'`

Question 12

A PySpark job reads a Delta table, applies a filter on a timestamp column, and then performs a window function partitioned by customer_id ordered by event_time. The job is slow, and the physical plan shows the filter is applied after the window. The developer wants the filter to reduce data before the window shuffle. Which action should the developer take?

Accepted Answer

Apply the filter on the timestamp column immediately after reading the Delta table, before calling the window function, so predicate pushdown and early filtering reduce rows entering the shuffle.. The physical plan shows the filter after the window, meaning the window shuffle processes all rows. Moving the filter to immediately after the read lets Spark push it into the Delta scan where possible and shrink the dataset before the window shuffle, reducing shuffle volume, memory use, and overall runtime.

Answer

Increase spark.sql.windowExec.buffer.spill.threshold so the window operator can spill more data to disk.

Answer

Repartition the DataFrame by customer_id before applying the filter so the window avoids a shuffle.

Answer

Cache the DataFrame before the window and call count() to materialize it, then apply the filter.

Question 13

You have a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. You need to ensure that the job processes each record exactly once, even after failures. Which of the following should you configure?

Accepted Answer

Set the `checkpointLocation` option to a reliable storage location.. Exactly-once processing in Structured Streaming relies on checkpointing to record progress and idempotent sinks to avoid duplicates. The checkpoint location stores offset information, allowing the query to resume without reprocessing. Delta Lake supports idempotent writes when used with checkpoints. Other options like Change Data Feed or manual deduplication do not provide the necessary fault tolerance.

Answer

Configure the Kafka source with `startingOffsets` set to `earliest`.

Answer

Enable idempotent writes by setting the Delta table property `delta.enableChangeDataFeed` to true.

Answer

Use `foreachBatch` to manually deduplicate records based on a unique key.

Question 14

You are monitoring a Structured Streaming query in Databricks and want to see the current status, including the number of input rows per second and the batch duration. Which of the following is the most direct way to access this information?

Accepted Answer

Use the `StreamingQuery.lastProgress` method on the query object.. The `StreamingQuery.lastProgress` method provides a structured snapshot of the most recent micro-batch's metrics, including input rate and batch duration. It is the direct API for monitoring. Log files are not structured, there is no SQL command for showing streaming queries, and Delta logs do not contain streaming metrics. Thus, `lastProgress` is the correct choice.

Answer

Query the `spark.streaming` log file on the driver node.

Answer

Run `spark.sql("SHOW STREAMING QUERIES")`.

Answer

Check the Delta table's transaction log for streaming metrics.

Question 15

A developer notices that a DataFrame transformation chain is executed twice: once for a count action used for logging and again for a write action. The source is a large Delta table and the repeated scan adds several minutes. Which action avoids the duplicate computation with the least risk?

Accepted Answer

Call cache or persist on the transformed DataFrame before the count so the second action reuses the materialized data.. When the same DataFrame lineage feeds two actions, Spark recomputes it for each action unless the intermediate result is materialized. Calling cache or persist after the expensive transformations stores the result so the later write reads it directly, removing the second full scan of the Delta table. The change is local, reversible, and does not alter results, making it the lowest-risk option among those presented.

Answer

Convert the DataFrame to a Pandas DataFrame for the count and then write from the original Spark DataFrame.

Answer

Set spark.sql.shuffle.partitions equal to the number of executor cores to speed up each execution.

Answer

Increase spark.sql.autoBroadcastJoinThreshold so more joins are broadcast and the plan becomes cheaper.