Databricks-Spark-Assoc Spark Architecture and Components — All Questions With Answers

Question 1mediummultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Based on the provided Spark configuration, what is the total amount of executor memory available to the application for data processing?

Exhibit

spark.executor.instances 4
spark.executor.cores 2
spark.executor.memory 4g

Question 2hardmulti select

Read the full Spark Architecture and Components explanation →

Which TWO of the following are true regarding the relationship between partitions and parallelism in Spark?

Question 3mediummultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Given the provided Spark configuration, what is the maximum number of concurrent tasks that can be processed at once across the entire cluster?

Exhibit

spark.executor.instances: 4
spark.executor.cores: 2
spark.executor.memory: 4g

Question 4mediummulti select

Read the full Spark Architecture and Components explanation →

Which THREE of the following are primary components of the Spark architecture's physical execution plan generation?

Question 5hardmulti select

Read the full Spark Architecture and Components explanation →

Which THREE of the following factors contribute to the total amount of memory available for Spark application tasks?

Question 6hardmulti select

Read the full Spark Architecture and Components explanation →

Which TWO of the following statements accurately describe the relationship between Spark Stages and Task execution?

Question 7mediummultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Based on the provided configuration, what is the size of the storage pool in bytes available for cached data on each executor?

Exhibit

spark.executor.memory 4g
spark.executor.cores 4
spark.memory.fraction 0.6
spark.memory.storageFraction 0.5

Question 8mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which property correctly defines the maximum number of cores an executor can use, and why is this setting important for Spark parallelism?

Question 9mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which component allows Spark to maintain fault tolerance by tracking the lineage of RDDs and recomputing lost partitions?

Question 10hardmultiple choice

Read the full Spark Architecture and Components explanation →

Which THREE components are provided by the Databricks control plane, distinct from the customer-managed compute cluster?

Question 11mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which architectural component is responsible for broadcasting variables from the Driver to all Executors?

Question 12mediummultiple choice

Read the full Spark Architecture and Components explanation →

A developer runs a Spark job on a Databricks cluster using spark-submit in local mode with the master URL set to local[4]. The job reads a large CSV file and performs a filter transformation. Which component is responsible for creating the logical plan for this job?

Question 13hardmulti select

Read the full Spark Architecture and Components explanation →

A Spark application is running on a Databricks cluster with adaptive query execution (AQE) enabled. The job reads a partitioned Parquet dataset, performs a join between two large tables, and then writes the result. The Spark UI shows that some tasks are taking significantly longer than others, and there is a high degree of skew in the join keys. Which TWO actions can help mitigate the skew and improve performance? (Choose two.)

Question 14mediummultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer is running a Spark application on a Databricks cluster with 4 worker nodes, each having 8 cores. The application reads a large Parquet file and performs a groupBy aggregation. The Spark UI shows that the job is divided into multiple stages. Which statement best describes the role of the DAG Scheduler in this scenario?

Question 15easymultiple choice

Read the full Spark Architecture and Components explanation →

A developer is writing a Spark application that uses a broadcast variable to distribute a small lookup table to all executors. The developer wants to ensure the broadcast variable is efficiently shared without being serialized with every task. Which component is responsible for distributing the broadcast variable from the Driver to the executors?

Question 16hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application reads a large Parquet dataset, performs a filter, and then writes the result to Delta Lake. The job fails with an OutOfMemoryError on the driver. The driver heap size is set to 4 GB, and the application uses a broadcast join for a small lookup table. Which action is most likely to resolve the driver OOM while preserving the broadcast join?

Question 17mediummulti select

Read the full Spark Architecture and Components explanation →

A Spark application running on Databricks uses a cluster with 3 worker nodes. Each worker node has 16 cores and 64 GB of memory. The application reads a large dataset and performs a join operation. Which TWO of the following statements accurately describe the role of the Task Scheduler in this scenario? (Choose two.)

Question 18hardmultiple choice

Read the full Spark Architecture and Components explanation →

A developer is running a Spark Structured Streaming job on Databricks that reads from a Delta table and writes to another Delta table with `foreachBatch`. The job uses `spark.sql.shuffle.partitions` set to 200. They notice that the driver memory is being exhausted during long-running streaming queries. Which Spark architecture component is most likely accumulating state and causing the driver memory issue?

Question 19easymultiple choice

Read the full Spark Architecture and Components explanation →

A developer is using Spark Structured Streaming to process a stream of JSON events from a Kafka topic. The developer notices that the streaming query's progress shows a growing number of tasks in the 'active' state, and the processing time per batch is increasing. The cluster has sufficient resources. What is the most likely cause of this behavior?

Question 20mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which component in the Spark architecture is responsible for maintaining the state of the Spark application and coordinating the execution of tasks across the cluster?

Question 21hardmulti select

Read the full Spark Architecture and Components explanation →

Which TWO of the following statements accurately describe the relationship between Spark Executors and memory management within a Databricks cluster?

Question 22easymultiple choice

Read the full Spark Architecture and Components explanation →

Which term describes the unit of work that is dispatched by the Driver to a specific Executor?

Question 23mediummultiple choice

Read the full Spark Architecture and Components explanation →

What is the primary function of the Spark DAG Scheduler?

Question 24mediummulti select

Read the full Spark Architecture and Components explanation →

Which THREE components are involved in the process of executing a Shuffle operation?

Question 25hardmultiple choice

Read the full Spark Architecture and Components explanation →

What is the consequence of having 'wide dependencies' in a Spark job regarding the Spark Architecture?

Question 26mediummultiple choice

Read the full Spark Architecture and Components explanation →

In the Databricks Spark environment, what is the role of the 'Shuffle Service'?

Question 27mediummultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Which of the following is the most likely cause for this 'shuffle fetch failure' in a Databricks cluster?

Exhibit

org.apache.spark.SparkException: Job aborted due to stage failure: Task ... in stage 1.0 (TID 1) had a shuffle fetch failure

Question 28hardmultiple choice

Read the full Spark Architecture and Components explanation →

Which property of RDDs (Resilient Distributed Datasets) is primarily responsible for Spark's fault tolerance during cluster execution?

Question 29easymultiple choice

Read the full Spark Architecture and Components explanation →

What happens when an action is called on a Spark DataFrame?

Question 30mediummultiple choice

Read the full Spark Architecture and Components explanation →

What is the primary benefit of the Catalyst Optimizer in the Spark SQL architecture?

Question 31mediummultiple choice

Read the full Spark Architecture and Components explanation →

What is the purpose of the 'Broadcast Variable' in the Spark architecture?

Question 32mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which component in the Spark architecture is responsible for scheduling tasks and managing the execution of jobs on the cluster?

Question 33hardmulti select

Read the full Spark Architecture and Components explanation →

Which TWO of the following statements accurately describe the role of the Spark Executor in a cluster deployment?

Question 34easymultiple choice

Read the full Spark Architecture and Components explanation →

Which of the following best describes the purpose of a Spark Session in a Databricks environment?

Question 35mediummultiple choice

Read the full Spark Architecture and Components explanation →

What happens when a Spark job triggers a 'shuffle' operation during execution?

Question 36hardmultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Which performance indicator suggests that Task 15 is likely causing a performance bottleneck during the execution of a join operation?

Exhibit

Task 12: 500MB (Local Disk)
Task 15: 1.2GB (Shuffle Read)
Task 22: 400MB (Shuffle Write)

Question 37easymultiple choice

Read the full Spark Architecture and Components explanation →

In the context of the Spark Driver, what is the 'DAG' and why is it important?

Question 38mediummultiple choice

Read the full Spark Architecture and Components explanation →

What is the primary role of the 'Cluster Manager' in Spark?

Question 39hardmultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Based on the error log, what is the most likely cause of the job failure?

Exhibit

org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 1.0 failed 4 times
... executor lost: ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits.

Question 40easymultiple choice

Read the full Spark Architecture and Components explanation →

Which of the following describes the 'Driver' process in a Spark application?

Question 41mediummultiple choice

Read the full Spark Architecture and Components explanation →

What is the function of the 'Executor' within the Spark execution model?

Question 42mediummultiple choice

Read the full Spark Architecture and Components explanation →

In the context of the Spark Driver, which component is specifically responsible for tracking the location of cached data blocks across the executors?

Question 43mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which Spark configuration property determines the maximum amount of memory the Spark Driver can request for itself when running on a Kubernetes cluster?

Question 44easymultiple choice

Read the full Spark Architecture and Components explanation →

Which component in the Spark cluster architecture is responsible for communicating directly with the Cluster Manager (e.g., YARN, Mesos, K8s) to request and release resources?

Question 45mediummulti select

Read the full Spark Architecture and Components explanation →

Which THREE components are part of the Spark execution environment that resides on the Driver node?

Question 46hardmultiple choice

Read the full Spark Architecture and Components explanation →

What is the primary function of the 'Shuffle Service' in a Spark cluster when using dynamic allocation?

Question 47easymultiple choice

Read the full Spark Architecture and Components explanation →

In Spark's cluster architecture, what happens to the tasks if the Driver node crashes during the execution of a job?

Question 48hardmultiple choice

Read the full Spark Architecture and Components explanation →

Refer to the exhibit. Which configuration issue is most likely causing this warning in a Spark application?

Exhibit

24/05/10 10:00:00 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources.

Question 49mediummulti select

Read the full Spark Architecture and Components explanation →

Which TWO factors influence the effective parallelism of a Spark application?

Question 50easymultiple choice

Read the full Spark Architecture and Components explanation →

What is the primary role of the 'Executor' process in the Spark distributed architecture?

Question 51mediummultiple choice

Read the full Spark Architecture and Components explanation →

In a Databricks Spark cluster, which component is primarily responsible for scheduling tasks and managing the distribution of computation across the worker nodes?

Question 52mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which TWO of the following statements correctly describe the role of the Spark Executor in a Databricks environment?

Question 53easymultiple choice

Read the full Spark Architecture and Components explanation →

Which Spark component is responsible for maintaining the Directed Acyclic Graph (DAG) of stages and tasks?

Question 54mediummultiple choice

Read the full Spark Architecture and Components explanation →

When a Spark application is running in Databricks, what determines the number of tasks that can run in parallel?

Question 55mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which process is responsible for tracking the location of data blocks cached in the executors?

Question 56hardmultiple choice

Read the full Spark Architecture and Components explanation →

In the context of Databricks, what occurs when a Spark stage is described as 'Shuffle-heavy'?

Question 57easymultiple choice

Read the full Spark Architecture and Components explanation →

Which component manages the lifecycle and allocation of executors in a Databricks cluster?

Question 58mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which TWO factors contribute to the 'Data Locality' optimization in Spark?

Question 59mediummultiple choice

Read the full Spark Architecture and Components explanation →

Which of the following correctly describes the relationship between a Spark Job and a Spark Stage?

Question 60mediummultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer submits a PySpark job that performs a wide transformation via a join operation across two large datasets. During execution, several tasks in the shuffle stage fail repeatedly due to transient network timeouts between worker nodes. How does Apache Spark's architecture handle these failed tasks?

Question 61mediummultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer submits a Spark application using spark-submit in client deploy mode from an edge node. The application reads a large Parquet dataset, performs a groupBy aggregation, and writes the result to a Delta table. The engineer notices that the Driver process runs on the edge node and remains alive throughout the application's lifetime. Which statement best describes the role of the Driver in this scenario?

Question 62mediummultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer runs a PySpark job on a Databricks cluster. The job reads a 500 GB Parquet dataset, applies a filter, and writes the result. The engineer notices that during execution, all tasks of a particular stage complete quickly except for a handful that take far longer, and the Spark UI shows these tasks are processing partitions that contain far more records than others. Which Spark architecture concept best explains this behavior, and what is the most appropriate remediation?

Question 63mediummultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer is tuning a Spark Structured Streaming job on Databricks that reads from a Kafka topic with 12 partitions. The job uses a static allocation of executors, each with 4 cores. The engineer notices that only 4 tasks are running concurrently, even though there are 12 Kafka partitions and 3 executors are available. Which Spark configuration is most likely causing this limitation?

Question 64hardmulti select

Read the full Spark Architecture and Components explanation →

A Databricks engineer is diagnosing why a Spark job's shuffle phase writes a very large amount of data to disk. The engineer wants to reduce shuffle overhead by changing how the job is structured and configured. Which TWO actions are most likely to reduce the volume of shuffle data written? (Choose two.)

Question 65easymultiple choice

Read the full Spark Architecture and Components explanation →

A developer submits a Spark application to a Databricks cluster using spark-submit with deploy mode set to cluster. During execution, one of the worker nodes hosting a task fails and is lost by the cluster manager. Which Spark component is responsible for rescheduling the failed task on another available executor?

Question 66hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark job reads a large Parquet dataset, performs a filter, and then a groupBy aggregation. The job's DAG shows two stages: one for the filter and one for the aggregation. The first stage has 200 tasks, and the second stage has 200 tasks. The job is running on a cluster with 10 executors, each with 8 cores. The engineer observes that the second stage takes significantly longer than the first. Which of the following is the most likely cause for the increased duration in the second stage?

Question 67easymultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application is running on a Databricks cluster with 3 worker nodes, each having 4 cores. The application uses the default configuration. How many tasks can run concurrently across the cluster?

Question 68easymultiple choice

Read the full Spark Architecture and Components explanation →

A developer submits a Spark application to a Databricks cluster. The application creates a SparkSession, reads a CSV file, and calls count() on the resulting DataFrame. Which component is responsible for translating this logical operation into a physical execution plan and coordinating its execution across the cluster?

Question 69mediummultiple choice

Read the full Spark Architecture and Components explanation →

A developer is tuning a Databricks job and wants to know how many tasks will be created for the final stage of a job that reads a Parquet file with 200 partitions, applies a filter, and then calls coalesce(10) before writing the result. Assuming no other repartitioning or shuffles occur, how many tasks will the final write stage contain?

Question 70easymultiple choice

Read the full Spark Architecture and Components explanation →

A developer is using Spark on Databricks and wants to monitor the progress of a job. They need to understand how the driver coordinates with executors. Which component is responsible for scheduling tasks onto executors and tracking their status?

Question 71mediummultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer runs a Spark job on a Databricks cluster using the default FIFO scheduler. They notice that a long-running job is holding all cluster resources, and short ad-hoc queries submitted later are stuck waiting. The engineer wants to allow concurrent scheduling of multiple jobs within the same Spark application so that short jobs can run while the long job is still executing. Which Spark configuration should be set to enable this behavior?

Question 72hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application running on a Databricks cluster uses a broadcast variable to distribute a small lookup table to all executors. During execution, the driver serializes the broadcast variable and sends it to each executor. Which component is responsible for storing the broadcast data on the executor side and making it available to tasks?

Question 73hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application running on Databricks uses broadcast joins for a small dimension table. A developer notices that the broadcast variable is not being sent to executors as expected, causing a shuffle instead. Which configuration property directly controls the maximum size of a table that Spark will automatically broadcast?

Question 74hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark job reads a large Parquet file, performs a groupBy operation, and then writes the result. During execution, the job fails with an OutOfMemoryError on the Driver. Which component is most likely responsible for the memory issue?

Question 75mediummulti select

Read the full Spark Architecture and Components explanation →

A Spark application is running on a cluster with 5 executors. The driver program creates a broadcast variable that is used in a transformation. Which two components are directly involved in distributing and using the broadcast variable? (Choose two.)

Question 76hardmultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer observes that a Spark Structured Streaming job on Databricks processes micro-batches with steadily increasing latency over several hours. The Spark UI shows that the number of active tasks per batch stays constant, but each task processes a growing amount of state. Which architectural behavior explains this pattern?

Question 77easymultiple choice

Read the full Spark Architecture and Components explanation →

A developer is writing a Spark application that will run on a Databricks cluster. They need to ensure that the driver program can communicate with the executors and that tasks are distributed correctly. Which component is responsible for coordinating the execution of tasks across the executors?

Question 78mediummulti select

Read the full Spark Architecture and Components explanation →

A developer is using Spark on Databricks and wants to understand how the Driver and Executors communicate during a job. Which two statements accurately describe this interaction? (Choose two.)

Question 79mediummultiple choice

Read the full Spark Architecture and Components explanation →

A developer is debugging a Spark job and observes that a particular stage has 200 tasks, but only 10 executors with 2 cores each are available. What will happen to the remaining tasks in that stage?

Question 80hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark job is running on Databricks and experiences a stage where tasks are taking much longer than expected. The Spark UI shows that some tasks have significantly higher shuffle read sizes than others, and the stage is skewed. Which Spark feature can automatically mitigate this skew by splitting large partitions into smaller ones?

Question 81mediummultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application is submitted to a Databricks cluster. The application uses a broadcast variable to distribute a small lookup table to all Executors. Which component is responsible for broadcasting this variable?

Question 82easymultiple choice

Read the full Spark Architecture and Components explanation →

In a Spark application running on a Databricks cluster, the driver program creates a SparkSession and defines a series of transformations. When an action is triggered, the driver requests resources from the cluster manager. Which component is responsible for negotiating and acquiring these resources on behalf of the Spark application?

Question 83mediummultiple choice

Read the full Spark Architecture and Components explanation →

A developer runs a Spark application on a Databricks cluster in Standard access mode. The application reads a Parquet file, applies a filter, and calls `df.cache()` before an action. During execution, the driver logs show that a stage is retried because a task failed with an executor lost error. Which component is responsible for rescheduling the failed task on another executor within the same application?

Question 84mediummultiple choice

Read the full Spark Architecture and Components explanation →

A Spark job reads a large CSV file, performs a groupBy aggregation, and then writes the result. The Spark UI shows that the job has multiple stages, and one stage has a large number of tasks. Which factor primarily determines the number of tasks in the stage that performs the aggregation?

Question 85easymultiple choice

Read the full Spark Architecture and Components explanation →

A data engineer is configuring a Spark application on Databricks. They set `spark.executor.instances` to 4, `spark.executor.cores` to 5, and `spark.executor.memory` to 16g. The cluster has 5 worker nodes, each with 16 cores and 64 GB RAM. What is the maximum number of tasks that can run concurrently across all executors?

Question 86hardmultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application is running in cluster mode on Databricks. The driver program is running on a worker node, and the application has been running for several hours. Suddenly, the driver node experiences a hardware failure and crashes. What happens to the running tasks and the application?

Question 87mediummulti select

Read the full Spark Architecture and Components explanation →

A developer is using Spark on Databricks and notices that a particular job has many stages due to shuffle operations. They want to understand the role of the shuffle in the Spark execution model. Which two statements accurately describe the behavior of a shuffle operation in Spark? (Choose two.)

Question 88mediummultiple choice

Read the full Spark Architecture and Components explanation →

A Spark application on Databricks uses a broadcast variable to distribute a large lookup table to all executors. The developer notices that the broadcast variable is not being used efficiently, as executors are still fetching the data multiple times. Which component is responsible for ensuring that the broadcast data is distributed only once per executor and cached there?

org.apache.spark.SparkException: Job aborted due to stage failure: Task 5 in stage 1.0 failed 4 times ... executor lost: ExecutorLostFailure (executor 2 exited caused by one of the running tasks) Reason: Container killed by YARN for exceeding memory limits.