Databricks-Spark-Assoc Practice Test 22 — 15 Questions

Question 1

Which of the following correctly describes the relationship between a Spark Job and a Spark Stage?

Accepted Answer

A stage is a set of parallel tasks that do not require a shuffle.. In Spark's execution architecture, a job is composed of multiple stages. Stages are defined by shuffle boundaries. When a job is submitted, the DAG scheduler breaks it into stages based on wide transformations that require data movement across the network. Understanding this hierarchy is essential for diagnosing performance issues, as it allows developers to identify which specific parts of their code lead to expensive shuffle operations, thus enabling better query optimization.

Answer

A job is a single task that runs on one executor.

Answer

Stages are independent and never depend on each other.

Answer

A job must contain exactly one stage.

Question 2

A developer is building a feature that must, for each `customer_id`, concatenate the distinct `product` values from many rows into a single comma-separated string. The result must contain each product only once per customer. Which single approach produces this result?

Accepted Answer

df.groupBy("customer_id").agg(concat_ws(",", collect_set("product")).alias("products")). Producing a distinct comma-separated list per customer requires an aggregation that deduplicates first. collect_set returns a set of unique product values, which concat_ws then joins into one string. collect_list keeps duplicates, and the other approaches either collapse to a single value or reshape the data into columns, none of which yield the required distinct concatenated string.

Answer

df.groupBy("customer_id").agg(concat_ws(",", collect_list("product")).alias("products"))

Answer

df.groupBy("customer_id").pivot("product").agg(count("product"))

Answer

df.select("customer_id", concat_ws(",", "product")).groupBy("customer_id").agg(first("product"))

Question 3

A Spark job reads a large Parquet dataset, performs a groupBy on a high-cardinality column, and writes the result to a Delta table. The job fails with a FetchFailedException on a particular executor. The Spark UI shows that the executor had sufficient memory but the shuffle fetch failed due to a connection reset. Which configuration change is most likely to resolve this issue?

Accepted Answer

Set spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait to higher values to handle transient network issues.. A FetchFailedException with connection reset during shuffle fetch is typically caused by transient network issues or shuffle service timeouts. Increasing spark.shuffle.io.maxRetries and spark.shuffle.io.retryWait makes the shuffle fetch more resilient by retrying failed attempts. This is the most direct configuration change to handle temporary network glitches without altering the overall job logic or resource allocation.

Answer

Increase spark.executor.memory to prevent the executor from being killed during shuffle.

Answer

Increase spark.reducer.maxSizeInFlight to allow larger shuffle blocks to be fetched.

Answer

Set spark.sql.adaptive.enabled=false to disable adaptive query execution and avoid shuffle re-computation.

Question 4

A data analyst is using the Pandas API on Spark to compute summary statistics. They call `psdf.describe()` on a large DataFrame and notice the job takes much longer than expected. They want to understand why this operation is more expensive than a similar operation on a small local pandas DataFrame. What is the primary reason?

Accepted Answer

`describe()` triggers a full scan and computes multiple aggregations that may require shuffling data across partitions.. `describe()` on a Pandas API on Spark DataFrame runs a distributed Spark job that scans all rows and computes multiple aggregates. Percentiles require approximate quantile algorithms that shuffle data across partitions. This is fundamentally more expensive than local pandas, which operates on in-memory data on a single machine without network shuffles.

Answer

`describe()` is a lazy operation that only builds a plan, so the delay is from plan construction rather than execution.

Answer

`describe()` caches the DataFrame in memory by default, and the caching step is what takes the extra time.

Answer

`describe()` converts the entire DataFrame to a local pandas DataFrame on the driver before computing statistics.

Question 5

You are processing a large, highly skewed PySpark DataFrame in Databricks and want to optimize a forthcoming join operation against a small lookup dimension table. Which TWO strategies are valid and effective DataFrame API techniques to optimize this join performance? (Choose TWO)

Accepted Answer

Use the broadcast() function on the small dimension table to distribute it to all worker nodes and avoid a shuffled hash join.. Optimizing joins involving skewed datasets requires avoiding shuffles for small tables or breaking up skewed keys. Broadcasting the small table eliminates the shuffle phase entirely, while salting keys distributes skewed join keys across multiple tasks, preventing memory bottlenecks on single executors during heavy shuffle exchanges.

Answer

Apply a broadcast join hint directly to the large skewed fact table to force executor nodes to cache its partitions in memory.

Answer

Increase the spark.sql.shuffle.partitions configuration to an extremely high number like 10000 to eliminate data skew entirely.

Answer

Convert the large DataFrame into a local Pandas DataFrame using toPandas() to perform the join operations locally on the driver node.

Question 6

A developer needs to join `orders` (large) with `customers` (small, a few thousand rows) on `customer_id`. They want to broadcast the small table and confirm the broadcast actually took effect. Which combination of actions is correct?

Accepted Answer

Call `orders.join(broadcast(customers), "customer_id")` and verify by inspecting the physical plan for `BroadcastHashJoin`. Broadcasting a small table avoids shuffling the large one, and the `broadcast()` hint is the explicit way to request it. Verification matters because hints can be ignored when the table exceeds the threshold or when the join type is unsupported. Inspecting the physical plan for `BroadcastHashJoin` is the standard confirmation, making the combination of hint plus plan inspection the correct answer.

Answer

Increase `spark.sql.autoBroadcastJoinThreshold` to a very large value and rely on the optimizer without checking the plan

Answer

Call `customers.repartition(1)` before the join so the small table is in one partition

Answer

Use `orders.join(customers.hint("shuffle_hash"), "customer_id")` and check the plan for `ShuffledHashJoin`

Question 7

A data engineer wants to run a Structured Streaming query that reads from a Kafka topic and writes aggregated counts to a console sink for debugging. The query uses a grouping aggregation on a tumbling event-time window. Which output mode must be used so that only rows that changed since the last trigger are emitted?

Accepted Answer

update. Update mode is designed to emit only rows whose aggregate values changed since the last trigger. This matches the debugging need to see evolving window counts without reprinting the full result each time. Append withholds results until windows finalize, complete reprints everything, and snapshot is not a supported mode, so update is the correct choice.

Answer

complete

Answer

snapshot

Answer

append

Question 8

Which component in the Spark architecture is responsible for scheduling tasks and managing the execution of jobs on the cluster?

Accepted Answer

Spark Driver. The Driver process is the central coordinator in Spark. It hosts the SparkContext, which converts user code into a Directed Acyclic Graph (DAG) and schedules tasks across the executors. Understanding this role is vital because it explains why the Driver can become a bottleneck if it handles too much data locally or manages excessive partitions, directly impacting the overall job latency and system stability in distributed environments.

Answer

Cluster Manager

Answer

Spark Executor

Answer

Storage Manager

Question 9

Which file format is best suited for performance-critical Spark applications that require efficient schema enforcement and column pruning?

Accepted Answer

Parquet. Parquet is a columnar storage format that natively supports predicate pushdown and column pruning, allowing Spark to read only the necessary data from disk. This drastically reduces I/O throughput and improves query performance significantly. Understanding why columnar formats are superior to row-based formats for analytics is a foundational concept for Databricks developers designing high-performance data lakes and efficient ETL pipelines at scale.

Answer

CSV

Answer

JSON

Answer

XML

Question 10

A developer notices that a PySpark DataFrame transformation chain runs a full scan of a Delta table each time a new action is invoked, even though the source data has not changed. The developer wants to persist the intermediate DataFrame in memory across actions. Which method should be used?

Accepted Answer

Call DataFrame.cache() before the first action so the DataFrame is stored in memory on first computation.. cache() persists the DataFrame across actions using the default MEMORY_AND_DISK level, so the first action computes and stores partitions and later actions read from the cache rather than rescanning the Delta table. Checkpointing writes to disk and cuts lineage but does not keep data in memory, while AQE and collect() do not provide reusable in-memory persistence.

Answer

Call DataFrame.collect() after each transformation to materialize the results on the driver.

Answer

Set spark.sql.adaptive.enabled to true so the optimizer reuses prior scan results automatically.

Answer

Call DataFrame.checkpoint() to truncate the lineage and write the data to a reliable file system.

Question 11

A data engineer is using Spark Connect to run a job on a Databricks cluster. They notice that when they call `df.count()`, the operation takes longer than expected. They suspect that the client is transferring data unnecessarily. Which statement best explains the data transfer behavior of `df.count()` in Spark Connect?

Accepted Answer

The server executes the count and returns only the count value to the client, minimizing data transfer.. In Spark Connect, actions like `count()` are executed on the server. The client sends the logical plan, and the server computes the result, returning only the scalar value. This minimizes data transfer and leverages the cluster's processing power. The client does not download data for aggregation.

Answer

The client sends a count request to the server, and the server returns a lazy evaluation plan that the client must execute.

Answer

The client sends the entire DataFrame to the server, which then counts the rows and returns the result.

Answer

The client downloads all rows to count them locally, then discards the data.

Question 12

What is the primary function of the 'Shuffle Service' in a Spark cluster when using dynamic allocation?

Accepted Answer

To allow executors to retrieve shuffle data from removed executors.. The External Shuffle Service allows executors to be decommissioned without losing shuffle files needed by downstream stages. In dynamic allocation, Spark frequently scales the number of executors based on workload. Without this service, if an executor holding shuffle map output files were terminated, the downstream tasks would fail because their input data would be permanently lost, forcing expensive recomputations of the upstream shuffle stages.

Answer

To compress shuffle data before it is written to the disk.

Answer

To rebalance data partitions across the cluster during a shuffle.

Answer

To increase the speed of network transfers during a shuffle.

Question 13

What is the purpose of the 'Broadcast Variable' in the Spark architecture?

Accepted Answer

To send a read-only variable to every node efficiently.. Broadcast variables allow the Driver to send a read-only copy of a large variable to every executor, rather than sending a copy with every single task. This drastically reduces network traffic and memory usage when joining small lookup tables with large datasets. Understanding this is critical for performance, as it prevents the redundant transmission of data and optimizes join operations, ensuring that the cluster remains efficient and avoids network congestion during large-scale operations.

Answer

To share mutable state across all executors.

Answer

To aggregate intermediate results from tasks.

Answer

To partition data for balanced shuffle operations.

Question 14

A developer is writing a Spark Connect application that runs on a local laptop and connects to a Databricks cluster. The application defines a Python function and registers it with `spark.udf.register` for use inside a `select` expression. When the code runs, the function executes on the server. Which statement describes how the UDF is handled in this scenario?

Accepted Answer

The UDF is serialized and shipped to the Spark Connect server, where it is deserialized and executed within the server's Python worker processes.. Spark Connect keeps the client thin by serializing operations, including UDF definitions, and sending them to the server for execution. The server deserializes the UDF and runs it in its Python worker processes, so distributed execution and data locality remain on the cluster while the developer keeps the familiar PySpark UDF API.

Answer

The UDF is rejected because Spark Connect does not support user-defined functions of any kind.

Answer

The UDF is converted into a SQL expression by the client and embedded directly in the query plan without server-side Python execution.

Answer

The UDF runs only on the client machine, and its outputs are transmitted back to the server as literal values.

Question 15

A data engineer is writing a PySpark job that must read Parquet data, apply transformations, and then write the result. They want to avoid recomputation if the resulting DataFrame is referenced multiple times in later stages. Which action best accomplishes this?

Accepted Answer

Call df.cache() or df.persist() before the DataFrame is reused. Marking the DataFrame with `cache` or `persist` stores its computed partitions so subsequent actions reuse them rather than replaying the full lineage. Forcing a count, converting to an RDD, or checkpointing without a configured directory either fails to store reusable data or introduces errors and overhead. Caching is the standard mechanism for avoiding recomputation across multiple downstream uses.

Answer

Call df.checkpoint() without setting a checkpoint directory

Answer

Call df.count() to force computation before reuse

Answer

Call df.rdd to convert the DataFrame to an RDD