Databricks-Spark-Assoc Spark Architecture and Components Practice Test 5 — 15 Questions

Question 1

A data engineer submits a Spark application using spark-submit in client deploy mode from an edge node. The application reads a large Parquet dataset, performs a groupBy aggregation, and writes the result to a Delta table. The engineer notices that the Driver process runs on the edge node and remains alive throughout the application's lifetime. Which statement best describes the role of the Driver in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The Driver schedules tasks, maintains the DAG, and coordinates with the cluster manager to allocate executors.. The Driver in client deploy mode runs on the submitting host and orchestrates the application: it builds the DAG, schedules stages and tasks, and communicates with the cluster manager to acquire executors. It does not execute data processing tasks or store data. Therefore, the statement that it schedules tasks and coordinates resource allocation is correct.

Answer

The Driver executes the actual data processing tasks and stores intermediate shuffle data on local disk.

Answer

The Driver is responsible for storing the final output data and serving it to downstream consumers.

Answer

The Driver acts as a passive monitor that only collects metrics and logs, while the cluster manager handles all scheduling.

Question 2

A data engineer runs a PySpark job on a Databricks cluster. The job reads a 500 GB Parquet dataset, applies a filter, and writes the result. The engineer notices that during execution, all tasks of a particular stage complete quickly except for a handful that take far longer, and the Spark UI shows these tasks are processing partitions that contain far more records than others. Which Spark architecture concept best explains this behavior, and what is the most appropriate remediation?

Accepted Answer

Data skew across partitions during a shuffle; the engineer should apply salting or repartitioning to distribute records more evenly.. Skewed partition sizes after a shuffle produce a few long-running tasks while most finish quickly, which matches the Spark UI pattern described. Redistributing records through salting or repartitioning balances the workload across tasks, addressing the root cause rather than masking symptoms with memory tuning or reduced parallelism.

Answer

Insufficient executor memory causing garbage collection pauses only on certain tasks; the engineer should increase spark.executor.memory.

Answer

The DAG Scheduler is serializing stages incorrectly; the engineer should disable adaptive query execution to force static stage boundaries.

Answer

Too few partitions in the source Parquet files; the engineer should call coalesce(1) before writing the output.

Question 3

A data engineer is tuning a Spark Structured Streaming job on Databricks that reads from a Kafka topic with 12 partitions. The job uses a static allocation of executors, each with 4 cores. The engineer notices that only 4 tasks are running concurrently, even though there are 12 Kafka partitions and 3 executors are available. Which Spark configuration is most likely causing this limitation?

Accepted Answer

spark.executor.instances is set to 1, so only one executor with 4 cores is available.. The concurrency of tasks in Spark is determined by the total number of cores available across all executors. If only one executor with 4 cores is allocated, only 4 tasks can run in parallel, regardless of the number of input partitions. The configuration spark.executor.instances controls how many executors are requested, and setting it to 1 restricts the job to a single executor's worth of cores.

Answer

spark.executor.cores is set to 4, limiting each executor to 4 concurrent tasks.

Answer

spark.default.parallelism is set to 4, overriding the number of partitions from Kafka.

Answer

spark.sql.shuffle.partitions is set to 4, limiting the number of tasks for shuffle operations.

Question 4

A Databricks engineer is diagnosing why a Spark job's shuffle phase writes a very large amount of data to disk. The engineer wants to reduce shuffle overhead by changing how the job is structured and configured. Which TWO actions are most likely to reduce the volume of shuffle data written? (Choose two.)

Accepted Answer

Pre-aggregate data with reduceByKey before a subsequent join so fewer records participate in the shuffle.. Eliminating a shuffle through broadcast hash join and reducing record counts through map-side pre-aggregation both cut the actual bytes that must be written and read across the network. Partition-count tuning, serializer choice, and caching change performance characteristics without removing the underlying data movement.

Answer

Increase spark.sql.shuffle.partitions to a much larger value without changing the query plan.

Answer

Call persist(MEMORY_ONLY) on the DataFrame before the shuffle stage.

Answer

Enable the Kryo serializer instead of the default Java serializer for the shuffle.

Question 5

A developer submits a Spark application to a Databricks cluster using spark-submit with deploy mode set to cluster. During execution, one of the worker nodes hosting a task fails and is lost by the cluster manager. Which Spark component is responsible for rescheduling the failed task on another available executor?

Accepted Answer

The Task Scheduler. When an executor is lost, the driver's Task Scheduler detects the failed tasks and relaunches them on other executors, honoring spark.task.maxFailures before failing the stage. The DAG Scheduler only regenerates stages when shuffle map outputs are lost. The cluster manager merely allocates resources and reports executor loss, and Catalyst is a compile-time optimizer with no runtime task responsibility.

Answer

The Catalyst Optimizer

Answer

The Cluster Manager

Answer

The DAG Scheduler

Question 6

A Spark job reads a large Parquet dataset, performs a filter, and then a groupBy aggregation. The job's DAG shows two stages: one for the filter and one for the aggregation. The first stage has 200 tasks, and the second stage has 200 tasks. The job is running on a cluster with 10 executors, each with 8 cores. The engineer observes that the second stage takes significantly longer than the first. Which of the following is the most likely cause for the increased duration in the second stage?

Accepted Answer

The second stage involves a shuffle, which requires data to be repartitioned across the network, causing additional I/O and serialization overhead.. The groupBy aggregation requires a shuffle, which redistributes data across the cluster based on the grouping key. This involves network transfer, disk I/O, and serialization/deserialization, all of which add significant overhead compared to narrow transformations like filter. Even with the same number of tasks, the shuffle makes the second stage slower.

Answer

The second stage is reading from disk, while the first stage reads from memory, causing slower performance.

Answer

The second stage has a larger number of partitions than the first stage, causing more tasks to be scheduled.

Answer

The second stage has more tasks than the first stage, leading to higher scheduling overhead.

Question 7

A Spark application is running on a Databricks cluster with 3 worker nodes, each having 4 cores. The application uses the default configuration. How many tasks can run concurrently across the cluster?

Accepted Answer

12. The maximum number of concurrent tasks in a Spark cluster is equal to the total number of cores available across all executors. In this scenario, with 3 worker nodes each having 4 cores, the total is 12 cores. Therefore, up to 12 tasks can run in parallel, assuming each task uses one core and there is no dynamic allocation or other constraints.

Answer

4

Answer

3

Answer

7

Question 8

A developer submits a Spark application to a Databricks cluster. The application creates a SparkSession, reads a CSV file, and calls count() on the resulting DataFrame. Which component is responsible for translating this logical operation into a physical execution plan and coordinating its execution across the cluster?

Accepted Answer

The Driver, which hosts the SparkSession and the DAG Scheduler.. The Driver is the control plane of a Spark application. It hosts the SparkSession, builds the logical and physical plans through the Catalyst optimizer, and uses the DAG Scheduler to break the plan into stages and tasks that executors run, then aggregates their results for the count().

Answer

The Cluster Manager, which allocates containers for the application.

Answer

The Executor processes running on worker nodes.

Answer

The Catalog, which stores table and column metadata.

Question 9

A developer is tuning a Databricks job and wants to know how many tasks will be created for the final stage of a job that reads a Parquet file with 200 partitions, applies a filter, and then calls coalesce(10) before writing the result. Assuming no other repartitioning or shuffles occur, how many tasks will the final write stage contain?

Accepted Answer

10 tasks, because coalesce(10) reduces the partition count to 10. coalesce(10) reduces the RDD from 200 partitions to 10 using narrow dependencies, so no shuffle occurs. Since the number of tasks in a stage equals the number of partitions in the RDD being processed, the final write stage launches 10 tasks. Filtering is also narrow and does not alter the partition count, so the coalesce result directly determines the task count.

Answer

1 task, because coalesce always merges everything into a single partition

Answer

200 tasks, because the filter forces a shuffle before coalesce

Answer

200 tasks, because coalesce does not change the number of partitions

Question 10

A developer is using Spark on Databricks and wants to monitor the progress of a job. They need to understand how the driver coordinates with executors. Which component is responsible for scheduling tasks onto executors and tracking their status?

Accepted Answer

The Task Scheduler. The Task Scheduler in Spark is responsible for assigning tasks to executors and monitoring their execution. It works closely with the DAG Scheduler, which breaks the job into stages, but the Task Scheduler handles the actual task-level scheduling and status tracking. This makes it the component that directly coordinates with executors.

Answer

The Cluster Manager

Answer

The DAG Scheduler

Answer

The Catalyst Optimizer

Question 11

A data engineer runs a Spark job on a Databricks cluster using the default FIFO scheduler. They notice that a long-running job is holding all cluster resources, and short ad-hoc queries submitted later are stuck waiting. The engineer wants to allow concurrent scheduling of multiple jobs within the same Spark application so that short jobs can run while the long job is still executing. Which Spark configuration should be set to enable this behavior?

Accepted Answer

spark.scheduler.mode=FAIR. The Fair Scheduler (spark.scheduler.mode=FAIR) enables multiple jobs within the same Spark application to share resources, so shorter jobs can run concurrently with a long-running job. The other options address resource scaling, registration timeouts, or straggler mitigation, none of which change intra-application job scheduling to allow concurrent execution.

Answer

spark.speculation=true

Answer

spark.dynamicAllocation.enabled=true

Answer

spark.scheduler.maxRegisteredResourcesWaitingTime=30s

Question 12

A Spark application running on a Databricks cluster uses a broadcast variable to distribute a small lookup table to all executors. During execution, the driver serializes the broadcast variable and sends it to each executor. Which component is responsible for storing the broadcast data on the executor side and making it available to tasks?

Accepted Answer

Block Manager. Broadcast variables are distributed to executors using a BitTorrent-like protocol, and each executor's Block Manager stores the data. The Block Manager caches the broadcast data in memory and makes it available to tasks. This avoids shipping the data with each task, reducing network overhead and memory usage. The Block Manager is integral to Spark's storage layer.

Answer

Task Scheduler

Answer

Shuffle Service

Answer

DAG Scheduler

Question 13

A Spark application running on Databricks uses broadcast joins for a small dimension table. A developer notices that the broadcast variable is not being sent to executors as expected, causing a shuffle instead. Which configuration property directly controls the maximum size of a table that Spark will automatically broadcast?

Accepted Answer

spark.sql.autoBroadcastJoinThreshold. The automatic selection of a broadcast join is governed by spark.sql.autoBroadcastJoinThreshold, which defaults to 10 MB in Spark. If the small table's estimated size is below this threshold, Spark broadcasts it; otherwise, it uses a shuffle join. Other properties like shuffle partitions, broadcast block size, and broadcast timeout affect execution details but not the decision to broadcast based on table size.

Answer

spark.broadcast.blockSize

Answer

spark.sql.broadcastTimeout

Answer

spark.sql.shuffle.partitions

Question 14

A Spark job reads a large Parquet file, performs a groupBy operation, and then writes the result. During execution, the job fails with an OutOfMemoryError on the Driver. Which component is most likely responsible for the memory issue?

Accepted Answer

The Driver. An OutOfMemoryError on the Driver typically occurs when the Driver attempts to hold too much data in memory. This can happen during actions like collect() or when broadcasting large variables. In this scenario, the groupBy operation might produce a large aggregated result that is being collected to the Driver, exceeding its memory. The Executors process data in a distributed manner, but the Driver aggregates final results, making it the likely culprit.

Answer

The DAG Scheduler

Answer

The Cluster Manager

Answer

The Executors

Question 15

A Spark application is running on a cluster with 5 executors. The driver program creates a broadcast variable that is used in a transformation. Which two components are directly involved in distributing and using the broadcast variable? (Choose two.)

Accepted Answer

The driver serializes the broadcast variable and sends it to each executor via a BitTorrent-like protocol.. Broadcast variables are distributed by the driver, which serializes and sends them to executors using an efficient broadcast protocol. Each executor then caches the variable and makes it available to all its tasks. This two-step process minimizes network traffic and ensures that the variable is easily accessible during task execution.

Answer

The DAG Scheduler ensures that broadcast variables are only used in narrow transformations.

Answer

The Cluster Manager replicates the broadcast variable across all nodes in the cluster.

Answer

The Task Scheduler assigns the broadcast variable to each task individually.