Databricks-Spark-Assoc Using Spark Connect Practice Question
A data engineer is building a Spark Connect application and wants to attach a small lookup table to every task without a shuffle. The table is 20 MB and the cluster has default settings. Which approach should the engineer use?
⚠ Common exam trap
The trap here is thinking that caching or repartitioning a small DataFrame causes a broadcast join, when only the broadcast hint or the auto broadcast threshold influences the join strategy chosen by the server optimizer.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Call `broadcast(lookup_df)` from `pyspark.sql.functions` and join it with the fact DataFrame.
To broadcast a small relation in Spark Connect, the developer uses the `broadcast` function from `pyspark.sql.functions` as a join hint. The hint is serialized into the logical plan and evaluated by the server-side optimizer, which decides to replicate the small side to all executors. Caching, repartitioning to one partition, or disabling the broadcast threshold do not produce a broadcast join and can add unnecessary shuffle or memory pressure.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Call `broadcast(lookup_df)` from `pyspark.sql.functions` and join it with the fact DataFrame.
Why this is correct
The `broadcast` hint tells the server-side optimizer to broadcast the small relation to all executors, avoiding a shuffle of the large fact table. Because the lookup is only 20 MB, it fits comfortably under the default auto broadcast join threshold, and the hint makes the intent explicit. This is the supported way to request a broadcast join in a Spark Connect application.
- ✗
Set `spark.sql.autoBroadcastJoinThreshold` to -1 and join normally.
Why it's wrong here
Setting the threshold to -1 disables automatic broadcast joins entirely, which is the opposite of what is needed. The optimizer would then select a shuffle-based join for the 20 MB table. This configuration change would prevent the desired broadcast behavior rather than enable it, so it fails the scenario's requirement of avoiding a shuffle.
- ✗
Call `lookup_df.cache()` and then join, relying on the cache to avoid the shuffle.
Why it's wrong here
Caching materializes the DataFrame in memory or on disk on the executors, but it does not change the join strategy. The optimizer may still choose a sort-merge join that shuffles both sides. Caching a 20 MB table is unnecessary and does not guarantee a broadcast; the join operator's physical implementation is decided independently of caching, so this does not meet the requirement.
- ✗
Use `lookup_df.repartition(1)` before the join to force a single partition.
Why it's wrong here
Repartitioning to one partition creates a shuffle and funnels all lookup rows through a single task, which can bottleneck the join and does not broadcast the data. The fact table would still need to be shuffled to align with that single partition. This approach increases shuffle cost and does not achieve the goal of attaching the lookup to every task without a shuffle.
About these practice questions
One of 295 original Databricks-Spark-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.