Databricks-DE-Pro Developing Code (Python/SQL) Practice Question
You have a large Spark DataFrame that you need to filter and save as multiple smaller Parquet files based on the values in a 'region' column. Which method should you use to optimize the file layout for subsequent queries?
⚠ Common exam trap
Candidates frequently confuse partitionBy with clustering or bucketing. They mistakenly believe that using partitionBy will automatically optimize all query types, ignoring that it creates small file problems if the cardinality is too high.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use the 'partitionBy' option in the write command.
Partitioning by a low-cardinality column like 'region' physically organizes data into folders. This allows engines to perform 'partition pruning', skipping irrelevant folders during query execution. This is a standard optimization in Databricks for any dataset with clear categorical groupings, as it directly reduces the I/O cost of queries that filter by the partition key, improving performance significantly for analysts and dashboard tools.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use the 'repartition' method on the DataFrame before writing.
Why it's wrong here
Repartitioning changes the number of partitions in memory, but it does not inherently create the folder structure needed for partition pruning. While it creates the correct number of files, it does not provide the same query-time benefits that partitioning by a specific column offers to the reader.
- ✓
Use the 'partitionBy' option in the write command.
Why this is correct
The 'partitionBy' command instructs Spark to organize the output into a directory structure based on the values in the specified columns. This allows downstream queries to use partition pruning to only scan the necessary folders, significantly reducing data read and increasing query efficiency for the region data.
- ✗
Use the 'sortWithinPartitions' method.
Why it's wrong here
Sorting within partitions is useful for efficient data reading and minimizing shuffle, but it does not create the physical folder structure required for partition pruning. Without 'partitionBy', the engine still has to inspect all files to determine if they contain the relevant region data, negating the pruning benefit.
- ✗
Use the 'coalesce' method.
Why it's wrong here
Coalesce is used to reduce the number of partitions in a DataFrame to avoid the overhead of shuffling. It is primarily used to merge small files before writing to the sink, but it does not provide any metadata benefits or folder-based optimization for subsequent query performance or data access.
About these practice questions
One of 267 original Databricks-DE-Pro practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-DE-Pro practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-DE-Pro exam.