Generative AI Leader Practice Question: Techniques to Improve Generative AI Model Output
You are a Generative AI architect at a large financial services firm. The firm has deployed a custom large language model (LLM) fine-tuned on proprietary financial reports to assist analysts in generating quarterly earnings summaries. The model is hosted on Vertex AI using a dedicated endpoint with autoscaling enabled. Recently, the model's output has exhibited two issues: (1) occasional factual inaccuracies about specific financial figures, and (2) a tendency to produce overly verbose and repetitive text in the summaries, sometimes exceeding the desired length of 200 words. The team has already tried adjusting the temperature parameter from 0.7 to 0.2 and increased the top-k sampling from 40 to 50, but the problems persist. The model's training data includes over 10,000 financial reports, and the fine-tuning process used low-rank adaptation (LoRA) with rank 16. The production environment uses a batch size of 1 for inference. You need to recommend a course of action that most directly addresses both the factual accuracy and verbosity issues without requiring a full retraining of the model. Which approach should you take?
⚠ Common exam trap
Google Cloud often tests the misconception that adjusting hyperparameters like temperature or top-k can fix factual accuracy and verbosity, when in reality these issues stem from the model's lack of external knowledge and lack of output constraints, which require architectural changes like RAG and constrained decoding.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Implement a retrieval-augmented generation (RAG) pipeline that queries a vector database of verified financial data, and apply constrained decoding with a maximum token limit and a repetition penalty.
It directly addresses both issues without retraining. A RAG pipeline grounds the model's outputs in verified financial data, eliminating factual inaccuracies. Constrained decoding with a maximum token limit and repetition penalty directly curbs verbosity and repetition, which temperature and top-k adjustments failed to fix.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase the LoRA rank to 32 and fine-tune the model for additional epochs on a curated subset of reports that focus on concise and accurate summaries.
Why it's wrong here
Increasing LoRA rank and fine-tuning further constitutes additional training, which the scenario explicitly excludes, and does not address verbosity directly. LoRA tuning is tempting as the standard adaptation method, and it would be correct when the model must absorb new domain behaviour rather than be steered at inference.
- ✓
Implement a retrieval-augmented generation (RAG) pipeline that queries a vector database of verified financial data, and apply constrained decoding with a maximum token limit and a repetition penalty.
Why this is correct
RAG grounds generation in verified figures retrieved from a vector database, correcting factual inaccuracies without retraining, while constrained decoding enforces the 200-word ceiling and penalises repetition, directly resolving verbosity. Temperature and top-k tuning cannot fix hallucinated figures or enforce length limits.
- ✗
Switch to a larger pre-trained model (e.g., PaLM 2 or GPT-4) and use the same fine-tuning data with higher rank LoRA to improve capability, then rely on the larger model's inherent accuracy.
Why it's wrong here
Swapping to a larger pre-trained model with higher-rank LoRA is effectively retraining and does not guarantee grounding in the proprietary figures. Larger models are tempting for general capability gains, and this would be right when the base model itself lacks reasoning capacity rather than retrieval grounding.
- ✗
Experiment with higher temperature (e.g., 0.9) and lower top-k (e.g., 20) to encourage more diverse and concise outputs, and add a post-processing step to truncate summaries to 200 words.
Why it's wrong here
Raising temperature increases randomness, worsening factual drift, while top-k and truncation only mask verbosity rather than reduce repetition. Sampling controls are tempting because they are quick tuning knobs, and they would suit creative-variety tuning, not grounded accuracy with length control.
Go deeper
Related to this question
About these practice questions
One of 1,008 original Generative AI Leader practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This Generative AI Leader practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Generative AI Leader exam.