Databricks-Spark-Assoc Spark Architecture and Components Practice Test 2 — 15 Questions

Question 1

A Spark application reads a large Parquet dataset, performs a filter, and then writes the result to Delta Lake. The job fails with an OutOfMemoryError on the driver. The driver heap size is set to 4 GB, and the application uses a broadcast join for a small lookup table. Which action is most likely to resolve the driver OOM while preserving the broadcast join?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Avoid using collect() or toPandas() on large datasets; write directly to Delta Lake.. Driver OOM in Spark is commonly caused by actions that bring large datasets to the driver, such as collect() or toPandas(). Writing directly to Delta Lake keeps processing distributed, avoiding driver memory overload while retaining the broadcast join. Increasing memory or changing join strategy does not address the underlying issue and may conflict with the requirement.

Answer

Replace the broadcast join with a sort-merge join.

Answer

Increase spark.driver.memory to 8 GB.

Answer

Set spark.sql.autoBroadcastJoinThreshold to -1 to disable broadcasting.

Question 2

A Spark application running on Databricks uses a cluster with 3 worker nodes. Each worker node has 16 cores and 64 GB of memory. The application reads a large dataset and performs a join operation. Which TWO of the following statements accurately describe the role of the Task Scheduler in this scenario? (Choose two.)

Accepted Answer

It launches tasks on executors based on data locality and available resources.. The Task Scheduler launches tasks on executors considering data locality and resource availability, and it retries failed tasks on other executors. These are core functions that ensure efficient and fault-tolerant execution. The DAG Scheduler handles stage creation, while the Catalyst Optimizer optimizes plans, and the cluster manager allocates resources.

Answer

It optimizes the physical plan by applying predicate pushdown.

Answer

It divides the job into stages based on shuffle dependencies.

Answer

It manages the allocation of executor memory and cores.

Question 3

A developer is running a Spark Structured Streaming job on Databricks that reads from a Delta table and writes to another Delta table with `foreachBatch`. The job uses `spark.sql.shuffle.partitions` set to 200. They notice that the driver memory is being exhausted during long-running streaming queries. Which Spark architecture component is most likely accumulating state and causing the driver memory issue?

Accepted Answer

The State Store for stateful operations. In Spark Structured Streaming, stateful operations rely on the State Store, which is managed by the driver and stores state across micro-batches. As the number of state keys grows, the State Store's memory footprint on the driver increases, potentially causing driver OOM. This is a known architectural consideration for long-running streaming jobs. The Block Manager, DAG Scheduler, and Catalyst Optimizer do not accumulate state in this manner.

Answer

The DAG Scheduler's stage tracking

Answer

The Catalyst Optimizer's plan cache

Answer

The Block Manager on executors

Question 4

A developer is using Spark Structured Streaming to process a stream of JSON events from a Kafka topic. The developer notices that the streaming query's progress shows a growing number of tasks in the 'active' state, and the processing time per batch is increasing. The cluster has sufficient resources. What is the most likely cause of this behavior?

Accepted Answer

The Kafka topic has a large number of partitions, and Spark creates one task per partition per batch.. In Spark Structured Streaming with Kafka, the number of tasks per micro-batch is directly tied to the number of Kafka partitions. Each partition becomes a Spark partition, and each Spark partition is processed by a task. If the Kafka topic has many partitions, the number of tasks per batch will be high, leading to increased processing time if resources are not scaled accordingly.

Answer

The watermark is not set, causing state to grow indefinitely and increasing the number of tasks.

Answer

The output mode is set to 'complete', causing the entire result table to be recomputed each batch.

Answer

The streaming query is using `foreachBatch` with a custom function that is not parallelized.

Question 5

Which component in the Spark architecture is responsible for maintaining the state of the Spark application and coordinating the execution of tasks across the cluster?

Accepted Answer

The Spark Driver. The Driver process is the central coordinator in Spark. It runs the main() method, creates the SparkContext, and performs RDD graph scheduling and task distribution. Understanding the Driver's role is crucial because it is the primary bottleneck for metadata operations and task scheduling in a Spark cluster, and failure here results in the loss of the application's state and active execution context.

Answer

The Cluster Manager

Answer

The Executor

Answer

The Spark Master

Question 6

Which TWO of the following statements accurately describe the relationship between Spark Executors and memory management within a Databricks cluster?

Accepted Answer

The storage memory region is primarily used for caching RDDs and DataFrames.. Executors manage memory through a unified memory manager, partitioning heap space between storage (caching) and execution (shuffles/joins). Understanding this architecture is vital because improper memory configuration leads to OOM errors or excessive spilling to disk. By balancing the memory pools dynamically, Spark avoids hard boundaries, allowing execution tasks to borrow space from storage when cache utilization is low, significantly improving overall job throughput.

Answer

Executors use fixed memory boundaries that cannot be adjusted during task execution.

Answer

Executors are allowed to access the Driver's memory pool to store large datasets.

Answer

Memory management is handled entirely by the Cluster Manager, not the Spark process.

Question 7

Which term describes the unit of work that is dispatched by the Driver to a specific Executor?

Accepted Answer

Task. A Task is the smallest unit of execution in Spark. The Driver takes a stage, splits it into multiple tasks based on the partitions of the data, and schedules them for parallel processing on executors. Knowing this is critical for performance tuning; if a job has too many tasks, the overhead of scheduling becomes significant, while too few tasks fail to leverage the available cluster parallelism.

Answer

Job

Answer

Stage

Answer

Executor

Question 8

What is the primary function of the Spark DAG Scheduler?

Accepted Answer

Converting logical transformation chains into stages of tasks.. The DAG Scheduler is responsible for translating the logical plan into a physical execution plan, specifically breaking the lineage into stages based on shuffle boundaries. By identifying wide dependencies that require data redistribution, it organizes the execution flow. This is fundamental for optimizing performance, as it minimizes data movement across the network by grouping together all narrow dependency transformations into a single executable stage before triggering a shuffle operation.

Answer

Managing low-level memory allocation for individual executors.

Answer

Resource negotiation with the Cluster Manager.

Answer

Serializing data for network transmission between workers.

Question 9

Which THREE components are involved in the process of executing a Shuffle operation?

Accepted Answer

Map-side output files on executor nodes.. Shuffles require the coordination of map-side output, network transfer, and reduce-side aggregation. Understanding these components is critical because shuffles are the most expensive part of a Spark job due to disk I/O and network latency. When data must be reshuffled, Spark must store map outputs, have the executors communicate to fetch this data, and then process the incoming streams to complete the final aggregation or join operations.

Answer

The Cluster Manager's central storage.

Answer

The Driver's task result accumulation.

Question 10

What is the consequence of having 'wide dependencies' in a Spark job regarding the Spark Architecture?

Accepted Answer

It forces the DAG scheduler to create a new stage boundary.. Wide dependencies occur when a partition in the parent RDD contributes to multiple partitions in the child RDD, necessitating a shuffle. This forces a boundary in the DAG, triggering the creation of a new stage. Understanding this is essential because every shuffle introduces network I/O, disk I/O, and serialization overhead, which are the primary performance costs that developers must minimize when designing efficient Spark transformations and data models.

Answer

It allows for task pipelining within a single stage.

Answer

It enables data locality optimizations automatically.

Answer

It reduces the total number of tasks in the job.

Question 11

In the Databricks Spark environment, what is the role of the 'Shuffle Service'?

Accepted Answer

It enables executors to fetch data from terminated executors.. The External Shuffle Service allows executors to be decommissioned or removed without losing intermediate shuffle data. By offloading shuffle file management to a persistent service, Spark ensures that if an executor terminates, other nodes can still fetch the data required to complete the shuffle. This is critical in Databricks for dynamic allocation and auto-scaling, as it maintains stability despite the frequent addition and removal of worker nodes during cluster runtime.

Answer

It manages the distribution of data across HDFS clusters.

Answer

It converts narrow transformations into wide ones.

Answer

It caches all RDD partitions in memory.

Question 12

Refer to the exhibit. Which of the following is the most likely cause for this 'shuffle fetch failure' in a Databricks cluster?

Accepted Answer

The map-side executor was terminated before the reduce-side task could fetch its data.. A shuffle fetch failure usually occurs when an executor loses the intermediate data that a downstream task is trying to read, often due to the executor being preempted or failing. In Databricks, this is a common symptom when autoscaling removes a node that held required shuffle files. Understanding this error is essential for debugging job stability and configuring cluster settings to ensure reliable data processing despite dynamic resource availability.

Answer

The Driver ran out of memory during task scheduling.

Answer

The transformation involves a narrow dependency.

Answer

The input data format is unsupported.

Question 13

Which property of RDDs (Resilient Distributed Datasets) is primarily responsible for Spark's fault tolerance during cluster execution?

Accepted Answer

The RDD Lineage graph.. Lineage (the dependency graph) is the core mechanism of RDD fault tolerance. Because RDDs are immutable and record their transformation history, Spark can reconstruct lost partitions by recomputing them from their parent RDDs. This is superior to traditional replication models because it avoids the high cost of copying data over the network, allowing Spark to maintain resilience while maximizing performance and minimizing storage overhead across the distributed cluster infrastructure.

Answer

Data replication across multiple nodes.

Answer

The Spark Driver's checkpointing of all intermediate tasks.

Answer

The use of a centralized data warehouse.

Question 14

What happens when an action is called on a Spark DataFrame?

Accepted Answer

The entire transformation graph is executed.. An action triggers the execution of the DAG. Spark creates a job, splits it into stages, and launches tasks on executors to process the data. Until an action is called, Spark only builds a logical plan (lazy evaluation). This is fundamental for query optimization; it allows the Catalyst Optimizer to analyze the entire plan before execution, ensuring that unnecessary operations are pruned and the most efficient physical plan is generated for the cluster.

Answer

The data is immediately written to disk.

Answer

The Spark context is shut down.

Answer

The cache is automatically cleared.

Question 15

What is the primary benefit of the Catalyst Optimizer in the Spark SQL architecture?

Accepted Answer

It automatically generates the most efficient execution plan.. Catalyst optimizes logical plans through rule-based and cost-based transformations, such as predicate pushdown and constant folding. By simplifying the query plan before it reaches the physical execution layer, it drastically reduces the amount of data processed. This is critical for performance because it minimizes I/O and CPU usage, ensuring that queries are executed using the most efficient physical operators possible, which is essential for scaling across large datasets.

Answer

It manages the physical cluster resources.

Answer

It serializes data for storage in parquet files.

Answer

It performs automatic garbage collection on executors.