Databricks-Spark-Assoc Practice Test 23 — 15 Questions

Question 1

You are designing a Structured Streaming job that must read from a file source and write to a Delta table. You want the job to be resilient to failures and to continue processing only new files after a restart. Which two actions should you take? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Configure a unique checkpoint location for the query.. Resilient incremental file ingestion requires two things: durable progress tracking and a sink that can commit batches atomically. A unique checkpoint location records which files and offsets have been processed, and a Delta sink provides transactional, idempotent writes that prevent duplicate effects when batches are retried. Throughput and ordering options do not affect recovery correctness.

Answer

Enable the option latestFirst on the file source.

Answer

Set the query to use trigger(once=True) so it stops after each run.

Answer

Set the file source option maxFilesPerTrigger to a high value to process all files at once.

Question 2

A Databricks job fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver for local processing. Which action should you take to resolve this?

Accepted Answer

Replace collect() with take(n) or write the DataFrame to storage.. The OutOfMemoryError on the driver is caused by collecting a large DataFrame. The correct fix is to avoid transferring all data to the driver by using take() for a sample or writing the results to storage. Increasing driver or executor memory or adjusting shuffle partitions does not address the fundamental problem of excessive data transfer to the driver.

Answer

Increase the executor memory.

Answer

Increase spark.driver.maxResultSize.

Answer

Set spark.sql.shuffle.partitions to a higher value.

Question 3

When a Spark job is stuck in a shuffle phase, what is the most effective first step to identify the root cause of the performance bottleneck?

Accepted Answer

Check the Spark UI 'Stages' tab for uneven task durations.. The Spark UI provides a detailed breakdown of stages, tasks, and memory usage. Examining the 'Stages' and 'SQL' tabs reveals which tasks are taking the longest, which is indicative of skew or resource contention. Learning to interpret the Spark UI is the single most important skill for a Databricks developer, as it turns opaque 'stuck' jobs into actionable data regarding partition distribution and executor behavior.

Answer

Restart the cluster to clear any cached data.

Answer

Increase the spark.executor.memory value.

Answer

Enable speculative execution immediately.

Question 4

You are analyzing a large dataset using Pandas API on Spark. You have a Pandas-on-Spark DataFrame `psdf` that was created from a Spark DataFrame with multiple partitions. You call `psdf.head(10)` to quickly inspect the data. What does this operation return?

Accepted Answer

A standard pandas DataFrame containing the first 10 rows.. The `head(n)` method in Pandas API on Spark returns a local pandas DataFrame containing the first n rows. It is intended for quick inspection of data, and it collects only the necessary rows to the driver. This behavior matches the pandas API, where `head` returns a DataFrame, but here it triggers a small action to bring data locally.

Answer

A Pandas-on-Spark DataFrame containing the first 10 rows.

Answer

A list of Row objects containing the first 10 rows.

Answer

A Spark DataFrame containing the first 10 rows.

Question 5

You are writing a Structured Streaming query that reads from a Kafka topic and outputs to the console. You want to see only the newly arrived data in each micro-batch, without aggregations. Which output mode should you use?

Accepted Answer

Append. In Structured Streaming, Append mode is the default and outputs only new rows added since the last trigger. For a stateless query like reading from Kafka and writing to console, each micro-batch contains only new data. Update and Complete modes are designed for aggregations, where they output updated or full results, respectively.

Answer

Update with watermark

Answer

Update

Answer

Complete

Question 6

A developer wants to start a Structured Streaming query that reads from a Delta table and writes to another Delta table, and needs the query to process all existing data in the source table on its first run. Which option should be set on the read stream?

Accepted Answer

No option is needed; the Delta source reads the initial snapshot by default on the first run when no checkpoint exists.. The Delta Lake streaming source is designed to read the full table snapshot as the first micro-batch when a query starts without a checkpoint, and then to continue with subsequent changes. No option is required to include existing data; the behavior is the default. Options like `startingVersion` and `ignoreChanges` alter other aspects of the read, not the initial snapshot inclusion.

Answer

`.option("ignoreChanges", "false")`

Answer

`.option("includeExistingData", "true")`

Answer

`.option("startingVersion", "0")`

Question 7

A data engineer observes that a Spark Structured Streaming job on Databricks processes micro-batches with steadily increasing latency over several hours. The Spark UI shows that the number of active tasks per batch stays constant, but each task processes a growing amount of state. Which architectural behavior explains this pattern?

Accepted Answer

Stateful operators maintain growing keyed state in the executors as more keys arrive, increasing per-task work.. Stateful streaming operators retain keyed state across micro-batches, and as distinct keys accumulate, each task must load, update, and write a larger state store. This raises per-task processing time while task parallelism stays fixed, which is exactly the pattern shown when active tasks are constant but per-task state keeps growing.

Answer

Broadcast variables are being re-sent to executors on every micro-batch, adding network overhead.

Answer

The Driver is accumulating unbounded metadata from the DAG Scheduler across batches.

Answer

The Cluster Manager is throttling executor allocation, reducing parallelism per batch.

Question 8

Which TWO factors contribute to the 'Data Locality' optimization in Spark?

Accepted Answer

The physical proximity of the storage to the compute nodes.. Data locality is a critical optimization where Spark tries to schedule tasks on the node where the data resides. This prevents massive data movement across the network, which is the slowest part of a distributed system. By understanding how Spark respects the location of data blocks during task scheduling, developers can better partition their data and choose optimal file storage layouts for their workloads.

Answer

The number of executors running on the driver node.

Answer

The version of the Spark driver being used.

Answer

The total amount of memory assigned to the driver.

Question 9

A job is reading a huge amount of data from a table, but only uses three columns. Which optimization technique will provide the most significant I/O performance benefit?

Accepted Answer

Column pruning.. Column pruning is the practice of reading only the required columns from a data source. In formats like Parquet, this reduces the total amount of data read from disk and transferred across the network. This is a crucial optimization for Databricks developers to reduce I/O bottlenecks and improve overall pipeline speed, especially when dealing with wide tables containing hundreds of unused columns in analytical query workloads.

Answer

Caching the DataFrame.

Answer

Increasing the number of partitions.

Answer

Broadcasting the table.

Question 10

A developer has a local Python script that connects to a Databricks cluster using Spark Connect and creates a DataFrame from a small list of tuples. They then call .collect() on the DataFrame and receive the results. Which statement accurately describes how the data and operations are processed in this scenario?

Accepted Answer

The client serializes the logical plan and sends it to the Spark server, which executes the plan and returns the collected results.. Spark Connect uses a client-server architecture where the client builds a logical plan and sends it to the Spark server via gRPC. The server executes the plan on the cluster and returns the results. This decouples the client from the Spark driver, allowing remote execution without a local Spark context.

Answer

The DataFrame is created and processed entirely on the client; the cluster is only used for storage.

Answer

The client executes the DataFrame operations locally using a built-in Spark engine, then syncs the results to the cluster.

Answer

The client sends the raw data to the cluster, which then returns a Python object that the client uses to perform further operations locally.

Question 11

A developer is building a Spark Connect client application in Python that runs on a local workstation and connects to a remote Databricks cluster. The application must construct a DataFrame from a list of Python dictionaries without requiring the data to be uploaded to cloud storage first. Which approach should the developer use?

Accepted Answer

Use spark.createDataFrame(list_of_dicts) directly, because Spark Connect serializes local Python collections into Arrow batches and sends them to the server.. Spark Connect allows client-side Python collections to be converted into DataFrames through the standard createDataFrame API. The client serializes the data into Arrow format and streams it to the server over gRPC, so the data need not be staged in cloud storage. Approaches relying on SparkContext, RDD parallelization, or broadcasting fail because Spark Connect deliberately omits the low-level RDD and context APIs.

Answer

Write the list of dictionaries to a local JSON file, then call spark.read.json() with the local file path.

Answer

Broadcast the list of dictionaries with spark.sparkContext.broadcast() and then call spark.createDataFrame() on the broadcast variable.

Answer

Use spark.sparkContext.parallelize(list_of_dicts) and convert the resulting RDD to a DataFrame with spark.createDataFrame().

Question 12

A developer has a Spark SQL DataFrame `df` with an array column named `scores` containing integers. They need to create a new column `passing` that is true only when every element in `scores` is greater than or equal to 70. Which Spark SQL higher-order function should they use?

Accepted Answer

forall(scores, s -> s >= 70). The forall higher-order function is designed to test whether every element in an array satisfies a given predicate, returning a single boolean. In this scenario, it correctly produces a column that is true only when all scores are 70 or higher. transform maps each element, filter selects elements, and exists checks for at least one match, none of which yield the required all-elements condition.

Answer

exists(scores, s -> s >= 70)

Answer

filter(scores, s -> s >= 70)

Answer

transform(scores, s -> s >= 70)

Question 13

Which THREE components are part of the Spark execution environment that resides on the Driver node?

Accepted Answer

DAGScheduler. The Spark Driver hosts key components for managing the cluster, including the DAGScheduler, which decomposes jobs into stages; the BlockManagerMaster, which manages metadata for cached blocks; and the TaskScheduler, which handles the execution of tasks. These components collectively ensure that the application logic is translated into a series of executable stages and that tasks are distributed efficiently, maintaining the central control architecture of a Spark application.

Answer

BlockManager

Answer

Executor Backend

Question 14

A Databricks Structured Streaming job reads JSON files from a cloud storage directory and writes aggregated results to a Delta table. The source directory receives new files continuously and files are never modified after being written. The pipeline must tolerate late-arriving event data by up to 30 minutes and must not reprocess already-emitted windows when the query is restarted. Which combination of configurations is required to meet these requirements?

Accepted Answer

Set a checkpoint location on a durable file system, apply `withWatermark("eventTime", "30 minutes")`, and use the `append` output mode with a Delta table sink.. The pipeline needs durable state recovery and late-data tolerance without re-emitting finalized windows. A checkpoint location on reliable storage ensures the query resumes from where it left off. A 30-minute watermark defines how long late events are accepted. The append output mode emits each window only once, after the watermark passes the window end, which matches the no-reprocessing requirement when writing to Delta.

Answer

Configure `spark.sql.streaming.checkpointLocation` to a reliable cloud storage path, use `withWatermark("eventTime", "30 minutes")`, and set the output mode to `update`.

Answer

Set the source option `maxFilesPerTrigger` to a high value, define a watermark of 30 minutes on the event-time column, and rely on the default checkpoint location provided by the Spark session.

Answer

Use `withWatermark("eventTime", "30 minutes")`, set the output mode to `complete`, and configure a checkpoint location on DBFS.

Question 15

A developer is using the Pandas API on Spark to process a large dataset. They need to apply a custom Python function to each value in a column. They consider using `psdf['col'].apply(custom_func)`. What should they be aware of regarding performance?

Accepted Answer

`apply` with a Python function is executed row-by-row in Python, which can be slow and may cause out-of-memory errors if the data is not partitioned well.. Using a Python function with `apply` in Pandas API on Spark processes data row-by-row in Python, which is slower than vectorized operations and can lead to memory issues if partitions are large. It is important to use built-in Pandas functions or Spark SQL functions when possible for better performance.

Answer

`apply` with a Python function is not supported in Pandas API on Spark and will raise a NotImplementedError.

Answer

`apply` with a Python function is automatically parallelized across all cores of the driver node, ensuring optimal performance.

Answer

`apply` with a Python function is executed in a vectorized manner using Arrow, so it is as fast as built-in Pandas functions.