Databricks-Spark-Assoc Using Spark Connect Practice Question
A developer runs a Spark Connect client session against a Databricks cluster with `spark.conf.set("spark.sql.shuffle.partitions", "400")`. The cluster is configured with 8 worker nodes. Which component actually applies the shuffle partition setting to the physical plan?
⚠ Common exam trap
The trap here is assuming the local Spark Connect client performs planning or optimization, when in fact it only constructs and transmits unresolved logical plans and configuration to the server.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
The Databricks cluster-side Spark driver, which builds and executes the physical plan.
Spark Connect splits responsibilities: the thin client builds unresolved logical plans and sends them, along with session configuration, to the server. The Databricks cluster-side Spark driver then analyzes, optimizes, and converts the plan into a physical plan, where settings such as the shuffle partition count influence exchange operators. Executors and the workspace control plane do not perform this planning step.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
The Databricks cluster-side Spark driver, which builds and executes the physical plan.
Why this is correct
With Spark Connect, the client sends unresolved logical plans and configuration over gRPC to the server, where the Spark driver performs analysis, optimization, and physical planning. The shuffle partition count is consumed during exchange planning on the driver, so the value takes effect against the 8-node cluster's execution. The client only declares the setting; the server enforces it.
- ✗
The local Spark Connect client process, which rewrites the plan before serialization.
Why it's wrong here
The local client serializes logical operations and configuration into protobuf and sends them to the server; it does not build or rewrite physical plans. Physical planning, including how many shuffle partitions are used by an exchange, happens on the server side where the Spark driver and cluster resources live. Therefore the client cannot apply the setting to the physical plan in this scenario.
- ✗
The Databricks workspace control plane, which injects the value into the cluster's Spark configuration at startup.
Why it's wrong here
The control plane manages cluster lifecycle, notebooks, and job metadata, but it does not inject per-session Spark SQL configuration values into a running Spark Connect session. Configuration set through the session is transmitted to the server and applied by the Spark driver, not by workspace management services. The control plane is therefore not the component that applies this value to the physical plan.
- ✗
The worker executors, which read the value from broadcast configuration during task scheduling.
Why it's wrong here
Executors receive tasks and shuffle files, but the number of shuffle partitions is decided during physical planning on the driver before tasks are scheduled. Executors do not choose partition counts or read session-level SQL configuration to alter planning. They simply process the partitions assigned to them, so they cannot be the component applying this setting here.
Visual reference
About these practice questions
This Databricks-Spark-Assoc question is part of Courseiva's 295-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.