Refer to the Exhibit Practice Questions

The driver node is experiencing a network latency issue.

Why it fails: Network latency between the driver and the cluster might cause intermittent issues, but ExecutorLostFailure is typically indicative of resource exhaustion on the worker node itself. It is a local failure of the executor process, not a network-level communication failure between the worker and the driver node.

The Spark job is attempting to use too many partitions.

Why it fails: While too many partitions can create metadata overhead, they do not inherently cause executor loss. Executor loss is almost always a result of the executor process crashing, which is usually triggered by memory pressure, not by the count of partitions created by the Spark scheduler.

The shuffle partition count is set too low for the data.

Why it fails: A low shuffle partition count might lead to slow tasks due to data skew, but it would not cause the executor process itself to be lost. The job would finish eventually, albeit slowly. Executor loss is a failure of the process itself, not a performance-related task completion issue.

The target Delta table default.sales contains corrupted parquet files that cause schema resolution errors on the remote server.

Why it fails: Corrupted parquet files would surface as analysis or IO exceptions during query execution, not a gRPC UNAVAILABLE at plan submission, which occurs before any file is read. Parquet corruption fits cases where the Connect session is healthy but table scans fail mid-execution.

The PySpark client library version installed locally is newer than the server-side Spark runtime version, causing protocol serialization mismatches.

Why it fails: Version skew produces serialisation or protocol errors, typically UNIMPLEMENTED or INTERNAL, not UNAVAILABLE, which signals the gRPC endpoint itself is unreachable. Client-server version mismatch fits environments where the Connect session establishes but individual RPC calls are rejected as unsupported.

The user running app.py lacks Hive metastore permissions to read the default database catalog on the Databricks workspace.

Why it fails: StatusCode.UNAVAILABLE is a gRPC transport failure indicating the Spark Connect server is unreachable, not a metastore authorisation denial, which surfaces as an analysis or permission exception. Metastore permission errors fit scenarios where the session connects successfully but catalog reads are rejected.

Increase the spark.driver.maxResultSize setting.

Why it fails: While this avoids the error, it is a band-aid that places massive memory pressure on the driver. The driver node is not designed to act as a storage or aggregation unit for huge datasets, and this approach will eventually lead to driver OOM errors as the data grows over time.

Disable the Spark broadcast join threshold.

Why it fails: Broadcast join settings have no relation to the result size returned to the driver. The error is caused by the final action returning data to the driver, not by join operations during the middle of the job's execution stages. This parameter change will not address the error at all.

Increase the number of executor instances.

Why it fails: Adding more executors might speed up the job, but it does not change the amount of data being returned to the driver. If the final result is large, you will still hit the driver result size limit regardless of how many executors were used to compute that final output data.

The partition size is too large; increase spark.sql.files.maxPartitionBytes.

Why it fails: This setting controls how much data a partition can hold when reading files. It has no relation to the task binary size, which is caused by objects captured within code closures. Increasing this value would only change the number of files processed, not the serialization behavior.

The executors have insufficient memory; increase spark.executor.memory.

Why it fails: While more memory might stop an OOM error, it does not fix the underlying inefficiency of sending large objects inside task binaries. The warning specifically points to a coding pattern that captures excessive data in closures, which should be refactored regardless of how much memory is available.

The cluster is out of network bandwidth; enable compression.

Why it fails: While compression can reduce network usage, it does not address the fundamental issue of capturing a large object within a closure. The warning is a symptom of poor code architecture where the developer is inadvertently causing large objects to be serialized and sent to every task.

The Driver ran out of memory during task scheduling.

Why it fails: If the Driver ran out of memory, the job would typically fail with an OOM error or a connection timeout, not a shuffle fetch failure. Shuffle fetch failures specifically point to missing map-side output files, which occur on worker nodes, not within the central Driver's memory space.

The transformation involves a narrow dependency.

Why it fails: Narrow dependencies do not involve shuffle fetch operations, as they do not require data exchange between executors. Therefore, a fetch failure cannot occur in a purely narrow transformation pipeline. Only shuffles induce the network interactions that result in fetch errors if the source data is unavailable.

The input data format is unsupported.

Why it fails: Unsupported data formats typically trigger read errors at the beginning of a job, not mid-shuffle. Shuffle fetch failures occur after successful map stages, implying the input was read correctly. Attributing this to the file format ignores the distributed nature of the error, leading to incorrect debugging paths.

The Spark cluster is configured with insufficient executor memory.

Why it fails: The error occurs at the driver level during the collection process, not the executor level. Executor memory manages the distributed transformation, while the driver is responsible for aggregating the final output of 'to_pandas'. Increasing executor memory will not solve a driver-side memory pressure issue.

The DataFrame contains too many columns to be processed.

Why it fails: While many columns increase the memory footprint, the error specifically mentions the DataFrame size being too large for a 'to_pandas' conversion. The limitation is about total volume of data exceeding the driver memory capacity, not the number of columns themselves or a structural limitation.

The DataFrame has not been properly cached before calling to_pandas.

Why it fails: Caching helps with performance if the data is reused, but it does not change the physical requirement of the driver to hold the entire dataset once 'to_pandas' is invoked. Caching is irrelevant here as it doesn't reduce the final memory footprint required on the driver node.

spark.driver.memory is set too high.

Why it fails: Driver memory affects the driver node, not the executor resource acquisition process. If the driver cannot start, the application fails entirely, rather than producing a warning in the TaskSchedulerImpl about the inability to accept resources for the initial job, which concerns workers.

The shuffle service is not enabled on the cluster.

Why it fails: While the shuffle service is important for performance and stability, its absence does not prevent the initial job from starting or acquiring resources. The application can run without the shuffle service, though it may be prone to failures during shuffle-heavy operations.

The number of partitions is set too high for the cluster.

Why it fails: High partition counts may lead to overhead, but they do not prevent the initial job from acquiring resources from the cluster manager. The warning specifically mentions that the job has not 'accepted any resources,' which points toward a resource negotiation issue, not a data processing volume issue.

The Spark Driver failed to schedule the tasks properly.

Why it fails: The Driver successfully scheduled the task, but the executor failed during execution due to memory constraints. The error message indicates that the failure occurred at the container level (YARN), which means the task was already running when it hit the memory limit, not that the scheduling itself was faulty.

There was a network failure between the Driver and the worker.

Why it fails: While 'ExecutorLostFailure' can sometimes be network-related, the specific reason provided—'exceeding memory limits'—rules out a simple network issue. The failure was caused by internal memory pressure on the executor container, which was detected and terminated by the resource manager to prevent node-level instability.

The cluster manager was unable to find available nodes.

Why it fails: The error log does not indicate an inability to provision resources; the executor was running before it was terminated. This means the resource manager had successfully allocated the container, but the application code subsequently requested more memory than the allocated container was allowed to use.

The executor nodes have insufficient memory for the shuffle operation.

Why it fails: Shuffle OOMs typically manifest as disk space errors or specific shuffle-related task failures, not a general Java heap space error during a collect operation. Since the error is explicitly linked to the collect action, the bottleneck is on the driver's heap, not the executor's shuffle memory buffers.

The cluster configuration uses too many small partitions.

Why it fails: Small partitions might lead to task scheduling overhead, but they do not cause a Java heap space error on the driver during a collect action. The error is strictly related to the volume of data being moved into memory, not the degree of parallelism defined by the partition count.

The broadcast join threshold is set too low for the current job.

Why it fails: Broadcast join issues typically occur during execution stages and would cause OOM errors on the executors, not the driver. Furthermore, these errors usually manifest as broadcast-related exceptions or OOM errors within the executor JVM, rather than the general heap space failure observed during a final collect action.

Task 12 having 500MB on Local Disk.

Why it fails: Disk usage is normal for tasks that spill data or manage intermediate results. Since 500MB is relatively small compared to the 1.2GB shuffle read seen in other tasks, this does not represent a significant bottleneck or an unusual distribution of work that would cause a major performance delay.

Task 22 having 400MB Shuffle Write.

Why it fails: Shuffle write represents data being output to the next stage. A value of 400MB is standard and expected for a partition's output. It does not indicate a bottleneck because it represents the completion of work rather than the reception of an overwhelming amount of data from other nodes.

The total volume across all tasks is too low.

Why it fails: The total volume is irrelevant to the existence of a bottleneck within a single task. A bottleneck occurs when work is unbalanced, not when the overall dataset is too small. Even with small datasets, a skewed partition will cause the specific task processing it to lag behind others.