Databricks-Spark-Assoc Developing DataFrame/DataSet API Applications Practice Question
You have a large DataFrame containing user transaction logs. You need to read this data and immediately repartition it by user_id to optimize downstream filtering operations. Which DataFrame API method should you use?
⚠ Common exam trap
Candidates often choose 'coalesce()' instead of 'repartition()' because they think it is faster, forgetting that 'coalesce' is designed to reduce partitions without a full shuffle, which can cause data skew.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
df.repartition('user_id')
The repartition method creates a new set of partitions across the cluster network, which helps distribute data evenly to prevent skew. This is a critical transformation in Spark to optimize shuffle performance for downstream queries and aggregations by ensuring balanced workloads across executors.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
df.coalesce('user_id')
Why it's wrong here
Coalesce is strictly designed to reduce the number of partitions locally without a full shuffle. Passing a column name to coalesce is invalid syntax because it only accepts an integer representing the target partition count, making it unsuitable for key-based distribution.
- ✓
df.repartition('user_id')
Why this is correct
Repartitioning by a specific column triggers a wide transformation that performs a full shuffle across the cluster. This groups all rows with the same user_id into the same partition, significantly improving the performance of subsequent filters and joins on that key.
- ✗
df.partitionBy('user_id')
Why it's wrong here
PartitionBy is a DataFrameWriter method used exclusively when writing data to disk to create directory-level partitioning structures. It does not exist as a transformation on standard DataFrames for in-memory repartitioning operations during active data manipulation pipelines.
- ✗
df.shuffle('user_id')
Why it's wrong here
There is no built-in method named shuffle directly available on the Spark DataFrame API. Spark manages shuffle operations automatically under the hood during wide transformations like repartition, join, and groupBy rather than exposing a direct shuffle transformation.
About these practice questions
Courseiva writes every Databricks-Spark-Assoc question from scratch — 295 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.