A data scientist notices that a text generation model deployed on Vertex AI returns repetitive outputs after a few turns in a chat application. What is the most likely cause and the best parameter adjustment?
Reducing top_p narrows the token pool, reducing repetition.
Why this answer
Repetitive outputs in a chat application after a few turns are typically caused by the model getting stuck in a loop due to high cumulative probability from top-p sampling. Reducing top_p limits the set of tokens considered at each step, forcing the model to explore less likely tokens and breaking the repetition cycle. This directly addresses the issue without sacrificing coherence, unlike temperature adjustments which affect randomness globally.
Exam trap
Google Cloud often tests the misconception that temperature and top-p both control randomness in the same way, but the trap here is that candidates confuse 'increasing randomness' (temperature) with 'limiting the sampling pool' (top-p), leading them to choose D instead of B.
How to eliminate wrong answers
Option A is wrong because max_output_tokens controls the length of the output, not the diversity of token choices; increasing it would allow longer repetitive sequences, not fix the repetition. Option C is wrong because overfitting is a training-phase issue unrelated to inference-time repetition; switching to a smaller model would reduce capacity but not specifically address the sampling behavior causing loops. Option D is wrong because increasing temperature adds randomness to all token probabilities, which can actually worsen repetition by making the model more likely to pick high-probability tokens repeatedly; the problem is too much diversity in the sampling set, not too little.