Databricks-ML-Assoc Model Development Practice Question

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

• ✓

  df.randomSplit([0.8, 0.2], seed=42)

  Why this is correct

  randomSplit splits a DataFrame into multiple DataFrames according to the provided weights, and the seed parameter ensures reproducibility. Using seed=42 guarantees that the same split is generated each time the code runs, which is essential for consistent model evaluation. This method is the standard PySpark approach for train-test splitting with a fixed random seed.

• ✗

  df.repartition(2)

  Why it's wrong here

  repartition changes the number of partitions in the DataFrame but does not split rows into training and testing sets. It redistributes data across partitions, which affects parallelism but not data splitting. It has no seed parameter for reproducibility of splits. Using it here would not achieve the desired train-test division, and it could cause data shuffling without providing a test set.

• ✗

  df.sample(fraction=0.8, seed=42)

  Why it's wrong here

  sample returns a random subset of the DataFrame but does not split it into complementary parts. It cannot produce both training and testing sets in one call, and the complement is not automatically generated. While it uses a seed, it is not designed for train-test splitting. The scientist would need additional logic to obtain the test set, making it less suitable than randomSplit.

• ✗

  df.cache()

  Why it's wrong here

  cache persists the DataFrame in memory to speed up subsequent operations but does not split data. It is an optimization technique, not a splitting method. It has no concept of random seeds or train-test division. Using cache would not produce separate training and testing sets, so it fails to meet the requirement of reproducible splitting.