Databricks-GenAI-Assoc Data Preparation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Run VACUUM with a retention of zero hours to delete old files and free space before the embedding job.

  Why it's wrong here

  VACUUM removes files no longer referenced by the transaction log; it does not merge the current small files into larger ones. A zero-hour retention also eliminates time travel history and can break concurrent readers, which is dangerous in production. Vacuuming reduces storage, not file count, so read throughput for the embedding job would not improve.

• ✗

  Increase the number of shuffle partitions to the maximum supported value so that each task writes a single small file.

  Why it's wrong here

  More shuffle partitions create more output files, which worsens fragmentation rather than reducing it. Writing one small file per task is the opposite of compaction and increases metadata and listing overhead for downstream readers. This setting is useful for large shuffles, not for consolidating an append-heavy table before a read-heavy embedding job.

• ✓

  Enable auto compaction and optimized writes on the Delta table so small writes are coalesced automatically.

  Why this is correct

  Auto compaction merges small files after a write, and optimized writes shuffle data so fewer, larger files are produced in the first place. Together they keep fragmentation low between manual maintenance runs, which suits a table receiving frequent small appends. This directly addresses the file-count problem before the embedding job reads the data.

• ✓

  Run OPTIMIZE on the table to compact small files, optionally with ZORDER on columns frequently used in filters.

  Why this is correct

  OPTIMIZE rewrites many small files into fewer, larger files, which reduces per-file overhead during the embedding job's scans. Adding ZORDER on filter columns clusters related data so predicate pushdown skips more files. For an append-heavy table, periodic compaction is the standard Delta Lake remedy before a large read-heavy workload such as embedding generation.

• ✗

  Convert the table to a Parquet directory and rely on the file system to merge small files during reads.

  Why it's wrong here

  Parquet directories do not merge files during reads, and converting away from Delta loses transaction logs, time travel, and the OPTIMIZE command itself. The file system lists and opens every small file, so read overhead persists. This change removes the very tooling that solves the fragmentation problem and adds no benefit for the embedding workload.