A healthcare startup has fine-tuned a Vertex AI PaLM 2 model on a dataset of medical records to generate patient summaries. The model produces fluent text but occasionally fabricates diagnoses not present in the input. The team has already tried increasing the training data size by 20% and adjusting the temperature from 0.7 to 0.2, but hallucinations persist. The summaries must be factually accurate for regulatory compliance. What should the team do next?
RAG provides grounded, up-to-date context, reducing hallucinations significantly.
Why this answer
Implementing a Retrieval-Augmented Generation (RAG) pipeline with Vertex AI Search grounds the model's output in retrieved, authoritative medical documents. This directly addresses the root cause of hallucination—the model's reliance on its parametric memory—by providing factual context at inference time, which is far more effective for regulatory compliance than adjusting generation parameters or training data size alone.
Exam trap
The trap here is that candidates often assume adjusting model parameters (temperature, tokens) or switching models will fix hallucinations, when in fact the core issue is the lack of external knowledge grounding, which only RAG or similar retrieval-based techniques can reliably address for factual accuracy.
How to eliminate wrong answers
Option A is wrong because increasing maximum output tokens does not improve factual accuracy; it only allows the model to generate longer text, which can actually increase the opportunity for hallucinations. Option C is wrong because adding more few-shot examples to the prompt does not prevent the model from fabricating diagnoses; few-shot learning guides style and format but does not ground the model in external, verifiable facts. Option D is wrong because switching the base model to Gemini 1.5 Pro without additional changes does not solve the hallucination problem; all large language models can fabricate information when relying solely on their training data, and the underlying issue of factual grounding remains unaddressed.