Scenario practice questions

The Driver executes the actual data processing tasks and stores intermediate shuffle data on local disk.

Why it fails: Data processing tasks are executed by executors, not the Driver. Executors read data, perform transformations, and write shuffle files to local disk. The Driver only coordinates and does not process the distributed data. In this scenario, the Driver on the edge node would not handle the groupBy aggregation directly.

The Driver is responsible for storing the final output data and serving it to downstream consumers.

Why it fails: The Driver does not store final output data; that is written by executors to the specified sink (e.g., Delta table). The Driver coordinates the write but does not hold the data. In this scenario, the output is written by executors, not the Driver on the edge node.

The Driver acts as a passive monitor that only collects metrics and logs, while the cluster manager handles all scheduling.

Why it fails: The Driver is not passive; it actively builds the execution plan, schedules tasks, and manages job progress. The cluster manager allocates resources but does not schedule Spark tasks. The Driver's role includes DAG scheduling and task coordination, which are essential for the application to run.

The cache is stored in the client process memory, allowing subsequent operations to avoid round trips to the server.

Why it fails: Spark Connect clients do not store cached data in local process memory. The client only holds a logical plan and session state. Caching occurs on the server, where the data resides. Subsequent operations still require communication with the server, though the server may avoid recomputation if the cache is used. The client does not cache results locally.

The cache is automatically persisted to cloud object storage and shared across all Spark Connect clients connected to the same cluster.

Why it fails: cache() does not automatically persist to cloud object storage, nor is it shared across clients. Caching is session-specific and stored in the cluster's memory or local disk. It is not a global shared cache. Other Spark Connect clients with separate sessions would not see the cached data unless they access the same session, which is not how caching works.

Calling cache() has no effect because Spark Connect does not support caching operations on remote DataFrames.

Why it fails: Spark Connect supports cache() on DataFrames. The operation is sent to the server and executed there. It is not ignored. The server will attempt to cache the data according to the storage level, just as with a classic SparkSession. The limitation is that the cache is remote, not that caching is unsupported.

4

Why it fails: This is the number of cores per worker node, but concurrency is aggregated across the cluster. The total number of cores across all nodes determines the maximum concurrent tasks. With 3 nodes, the total is 12 cores, so 12 tasks can run in parallel, not just 4.

3

Why it fails: This would be the number of worker nodes, but concurrency is determined by the total number of cores available across all executors. Each core can run one task at a time. With 3 nodes and 4 cores each, there are 12 cores, so 12 tasks can run concurrently, not 3.

7

Why it fails: This number does not correspond to any obvious default. It might be a miscalculation, such as adding nodes and cores (3+4=7), but concurrency is based on the product of nodes and cores per node, not the sum. The correct number is 12.

readings.collect().map(lambda r: (r.sensor_id, r.celsius * 9/5 + 32))

Why it fails: collect() materializes every row on the driver, which is exactly what the scenario says to avoid and will fail or exhaust memory on a large DataFrame. The subsequent Python map() operates on a plain list, not a distributed DataFrame, so it does not produce a Spark DataFrame at all. This option violates both the distribution and schema requirements.

readings.rdd.map(lambda r: (r.sensor_id, r.celsius * 9/5 + 32)).toDF(["sensor_id", "fahrenheit"])

Why it fails: Converting to the RDD API discards the Catalyst optimizer, Tungsten code generation, and schema metadata that a DataFrame expression would retain. The map(lambda ...) also runs Python row-by-row, which is far slower than a native column expression, and toDF() must re-infer or restate the schema. It does avoid driver collection, but it is not the operation the scenario is asking for and loses DataFrame optimizations.

readings.withColumn("fahrenheit", col("celsius") * 9/5 + 32).drop("celsius")

Why it fails: withColumn() would correctly add the fahrenheit column, but the scenario explicitly requires keeping the original sensor_id and producing fahrenheit; dropping celsius is not requested and changes the output schema. If the intent were only to add a column, withColumn() would be acceptable, but the drop() makes this option fail the stated requirement of preserving the input columns besides the derived one.

left_df.join(right_df, on="id", how="right_outer")

Why it fails: A right outer join preserves all rows from right_df, which is the opposite of the requirement. Rows in left_df without a match would be dropped, and unmatched right rows would be retained with null left columns, inverting the desired behavior.

left_df.join(right_df, on="id", how="inner")

Why it fails: An inner join returns only rows where the key exists in both DataFrames, discarding any left rows without a match. This contradicts the requirement to retain all rows from left_df, so the result would silently drop unmatched left records.

left_df.join(right_df, on="id", how="full_outer")

Why it fails: A full outer join keeps unmatched rows from both sides, so right_df rows with no counterpart in left_df would also appear. The scenario only requires preserving left rows and matching right rows, so full_outer introduces extra rows that are not wanted.

Increase spark.sql.files.maxPartitionBytes to read more data per task.

Why it fails: Increasing maxPartitionBytes may reduce the number of partitions, but it does not solve the small files issue; each small file still requires a separate read operation. This setting controls how data is split into partitions, not how files are organized. It could even lead to larger partitions that are less efficient if the files are small.

Set spark.sql.autoBroadcastJoinThreshold to -1 to disable broadcast joins.

Why it fails: Disabling broadcast joins has no impact on reading small files from a Delta table. This setting is relevant for join strategies, not file scanning. The scenario involves a filter and aggregation, not a join, so this change would not affect read performance. It is unrelated to the small files problem.

Enable Databricks Delta Cache to cache the table files on local SSDs.

Why it fails: Delta Cache can speed up repeated reads of the same data, but it does not address the underlying issue of many small files. The initial read will still suffer from the overhead of opening numerous files. The filter and aggregation would benefit more from reducing file count or skipping data, which Delta Cache alone does not provide.

df.select("status").distinct()

Why it fails: distinct() performs a shuffle to deduplicate values and returns only the distinct status strings, not the original rows. It would not filter to active records in the sense of preserving row-level data, and it discards all other columns. This option changes both the shape and the meaning of the result, so it does not satisfy keeping active rows.

df.groupBy("status").count().filter(col("status") == "active")

Why it fails: groupBy triggers a shuffle to bring rows with the same status together, which the scenario explicitly wants to avoid. The output is also a count per status rather than the original rows, so the schema and semantics differ from simply selecting active records. It returns one row for active instead of every active record, which does not meet the requirement.

df.orderBy("status").filter(col("status") == "active")

Why it fails: orderBy induces a full range shuffle to sort rows, which contradicts the requirement to avoid a shuffle and is pointless because the subsequent filter discards everything that is not active. The final result would be correct rows in an arbitrary order, but the sort cost is wasted. A simple where() predicate achieves the same output far more cheaply.

The checkpoint location caches the entire streamed DataFrame in memory, allowing faster recovery after a driver restart.

Why it fails: Checkpoints do not cache DataFrames in memory. They store metadata and state, not the actual data rows. Caching a streaming DataFrame is not supported for recovery purposes, and memory is not durable across restarts. This option misrepresents the purpose of checkpointing.

The checkpoint location stores a copy of the Kafka topic's data, enabling replay from the checkpoint instead of Kafka.

Why it fails: Checkpoints do not store the source data itself. They store offsets and state. Replay still requires the original source (e.g., Kafka) to be available. This option incorrectly suggests that checkpointing duplicates the source data, which is not how Structured Streaming achieves fault tolerance.

The checkpoint location automatically deduplicates records in the Delta table by comparing primary keys.

Why it fails: Deduplication is not automatic based on primary keys. While Delta Lake supports MERGE for upserts, Structured Streaming does not infer primary keys or deduplicate without explicit logic. The checkpoint tracks progress, not record uniqueness. This option confuses checkpointing with data deduplication features.

The DAG Scheduler

Why it fails: The DAG Scheduler is a component within the Driver that plans stages and tasks. It does not directly consume large amounts of memory for data; its memory usage is for scheduling metadata. An OutOfMemoryError due to data size is not caused by the DAG Scheduler's operations, but rather by the Driver's data handling, such as collecting results.

The Cluster Manager

Why it fails: The Cluster Manager allocates resources and launches executors but does not process data or hold job results. It is not involved in memory management for data processing. An OutOfMemoryError on the Driver is unrelated to the Cluster Manager's operations, which are limited to resource negotiation and executor lifecycle.

The Executors

Why it fails: Executors handle the actual data processing and caching. An OutOfMemoryError on the Driver indicates the Driver's memory is exhausted, not the Executors'. While Executors can also run out of memory, the error specifically mentions the Driver. The Driver collects results and manages job scheduling, so memory issues there often stem from actions like collect() or large broadcast variables.

The UDF is rejected because Spark Connect does not support user-defined functions of any kind.

Why it fails: Spark Connect does support UDFs through serialization to the server. While some client-side APIs are restricted, UDFs are explicitly supported by sending their definitions to the server. Claiming a blanket prohibition is incorrect for this scenario.

The UDF is converted into a SQL expression by the client and embedded directly in the query plan without server-side Python execution.

Why it fails: The client does not translate arbitrary Python UDF logic into SQL. Spark Connect sends the serialized UDF to the server, which executes it in Python workers. There is no automatic conversion of Python code into equivalent SQL expressions in this flow.

The UDF runs only on the client machine, and its outputs are transmitted back to the server as literal values.

Why it fails: Spark Connect does not execute UDFs on the client and ship literal results. That would defeat distributed processing and break on large datasets. The server owns execution, so the function must be serialized and sent to the server, not evaluated locally on the laptop.

The DataFrame is created and processed entirely on the client; the cluster is only used for storage.

Why it fails: This is incorrect because Spark Connect follows a client-server architecture where the client sends logical plans to the server for execution. Creating a DataFrame from local data still results in a logical plan that is executed on the Spark server. The client does not process the DataFrame operations locally; it only builds the plan and collects results.

The client executes the DataFrame operations locally using a built-in Spark engine, then syncs the results to the cluster.

Why it fails: This is incorrect because Spark Connect does not run a local Spark engine on the client. The client only constructs logical plans and communicates with the remote Spark server. There is no local execution of DataFrame operations; all computation occurs on the cluster.

The client sends the raw data to the cluster, which then returns a Python object that the client uses to perform further operations locally.

Why it fails: This is incorrect because the client does not send raw data for local processing. Instead, it sends the logical plan representing the operations. The cluster executes the plan and returns only the final results when an action is triggered. The client does not perform further distributed operations locally.

exists(scores, s -> s >= 70)

Why it fails: The exists higher-order function returns true if at least one element satisfies the predicate. It would return true if any score is 70 or above, not if all are. This is the logical OR across elements, whereas the requirement is a logical AND across elements. Using exists would incorrectly mark rows as passing when only some scores meet the threshold.

filter(scores, s -> s >= 70)

Why it fails: The filter higher-order function returns a new array containing only the elements that satisfy the predicate. It does not return a boolean; it would give an array of the passing scores. To determine if all elements pass, you would need to compare the size of the filtered array to the original, which is not what the question asks for.

transform(scores, s -> s >= 70)

Why it fails: The transform function applies a lambda to each element and returns a new array of the same length, so it would produce an array of booleans, not a single boolean. To get a single true or false, you need a reduction-style higher-order function like exists or forall. transform is for element-wise mapping, not for aggregating an array into a scalar condition.