A company is deploying a Gemini 1.0 Ultra model for a code generation assistant. They have set up Vertex AI Model Evaluation with a custom evaluation dataset to measure pass@1 accuracy. The initial evaluation shows 65% pass@1. They want to improve to 80% without collecting more training data. They have already attempted basic prompt engineering (e.g., 'write correct code') with limited improvement. Which approach is most likely to achieve the desired improvement?
Chain-of-thought elicits reasoning steps, improving accuracy beyond basic prompting.
Why this answer
Chain-of-thought prompting with few-shot examples is the most effective approach because it guides the model through step-by-step reasoning, which is critical for complex code generation tasks. This technique leverages the model's in-context learning ability to improve accuracy without additional training data, directly addressing the need to boost pass@1 from 65% to 80%.
Exam trap
Google Cloud often tests the misconception that hyperparameter tuning (like temperature or beam search) can substitute for structured prompting techniques, when in reality, chain-of-thought prompting directly addresses the reasoning gap that limits pass@1 accuracy in code generation.
How to eliminate wrong answers
Option A is wrong because reducing temperature to 0 and setting top_p to 1 makes the model deterministic, which may reduce diversity but does not inherently improve correctness for complex code generation; it can even cause repetitive or suboptimal outputs. Option B is wrong because increasing output tokens and enabling beam search with width 4 can improve exploration but does not guarantee higher pass@1 accuracy; beam search is more suited for tasks like translation and may not align with the goal of generating a single correct code snippet. Option D is wrong because applying RLHF requires a reward model trained on human preferences, not just the existing evaluation dataset, and this approach demands significant additional data and computational resources, contradicting the constraint of not collecting more training data.