Databricks-GenAI-Assoc Data Preparation Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✗

  Load the table into a pandas DataFrame on the driver, apply filtering, and write it back using the Databricks SQL connector row by row.

  Why it's wrong here

  Collecting the entire reviews table to a pandas DataFrame on the driver will exhaust memory for any non-trivial dataset and forces single-node processing. Row-by-row writes through the SQL connector are extremely slow and stress the warehouse. This pattern does not scale and would fail in production.

• ✗

  Create a view that filters short and spam reviews, then point the fine-tuning job at the view without materializing a new table.

  Why it's wrong here

  A view re-evaluates filters every time it is queried, so the fine-tuning job would recompute token counts and spam checks on each run. Materializing the cleaned dataset avoids this repeated cost and gives a stable snapshot for training reproducibility. Views are useful for exploration but not for expensive preprocessing consumed by training.

• ✗

  Use Delta Live Tables with a streaming table that ingests all reviews and applies expectations to drop short and spam rows at write time.

  Why it's wrong here

  Delta Live Tables expectations can drop or quarantine rows, but this scenario is a one-time batch preparation of existing data, not a streaming ingestion. Setting up a streaming pipeline adds unnecessary complexity and does not match the batch nature of the fine-tuning dataset preparation. Expectations also do not compute token counts unless a UDF is defined.

• ✓

  Use a Spark DataFrame with a token-count UDF and a filter on the spam flag, then write the result to a Unity Catalog table with .write.mode("overwrite").saveAsTable().

  Why this is correct

  Filtering with Spark transformations keeps the work distributed and lets Catalyst optimize the plan. A token-count UDF computes length per row, and a simple filter removes spam-flagged rows. Writing with saveAsTable into Unity Catalog creates a governed training table that downstream fine-tuning jobs can read with lineage intact.