Databricks-Spark-Assoc Using Spark SQL Practice Question
You are processing a large dataset in Spark SQL and need to ensure that small files are avoided when writing data to Delta Lake. Which approach effectively minimizes small file generation during write operations?
⚠ Common exam trap
Candidates often suggest manual partitioning or repartitioning before every write. They fail to recognize that enabling built-in Delta Lake features like Auto Optimize is the more efficient, automated solution.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Enable 'autoOptimize' and 'optimizeWrite' at the Delta table level.
Optimizing file sizes is crucial for read performance and metadata management in Delta Lake. Using the OPTIMIZE command or enabling Auto Optimize are the standard patterns to address the small file problem. These techniques consolidate fragmented data into larger, performant files, reducing the overhead on the query engine and preventing performance degradation during subsequent read operations. This is a fundamental skill for maintaining healthy, scalable data lakes on Databricks.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Execute a DROP TABLE command before overwriting the existing table every time.
Why it's wrong here
Dropping and recreating tables is inefficient and causes significant metadata overhead. It does not address small file generation and can lead to data loss or downtime. Instead, use overwrites or merge operations with native optimization features designed to handle file compaction and layout management automatically.
- ✗
Increase the spark.sql.shuffle.partitions configuration to a very high value.
Why it's wrong here
Increasing shuffle partitions primarily affects join and aggregation operations rather than file write sizes. While it might lead to more tasks, it often results in many smaller files if the input data volume per task is too low, exacerbating the small file problem rather than solving it.
- ✓
Enable 'autoOptimize' and 'optimizeWrite' at the Delta table level.
Why this is correct
Enabling these properties allows Databricks to automatically coalesce small writes into larger files during the write operation itself. This significantly reduces the number of small files created by concurrent or frequent streaming writes, ensuring that data is laid out optimally for future analytical queries without manual intervention.
- ✗
Use the 'repartition(1)' method on the DataFrame before writing to storage.
Why it's wrong here
Using repartition(1) forces all data through a single executor, creating a massive bottleneck that breaks parallel processing. While it produces a single file, it creates a severe performance degradation for large datasets and does not scale horizontally, making it an inappropriate solution for production Spark SQL pipelines.
About these practice questions
One of 295 original Databricks-Spark-Assoc practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official Databricks exam blueprint
This Databricks-Spark-Assoc practice question is part of Courseiva's free Databricks certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Databricks-Spark-Assoc exam.