Generative AI Leader Practice Question: Techniques to Improve Generative AI Model Output
You are a Generative AI architect at a large financial services firm. The firm has deployed a custom large language model (LLM) fine-tuned on proprietary financial reports to assist analysts in generating quarterly earnings summaries. The model is hosted on Vertex AI using a dedicated endpoint with autoscaling enabled. Recently, the model's output has exhibited two issues: (1) occasional factual inaccuracies about specific financial figures, and (2) a tendency to produce overly verbose and repetitive text in the summaries, sometimes exceeding the desired length of 200 words. The team has already tried adjusting the temperature parameter from 0.7 to 0.2 and increased the top-k sampling from 40 to 50, but the problems persist. The model's training data includes over 10,000 financial reports, and the fine-tuning process used low-rank adaptation (LoRA) with rank 16. The production environment uses a batch size of 1 for inference. You need to recommend a course of action that most directly addresses both the factual accuracy and verbosity issues without requiring a full retraining of the model. Which approach should you take?
⚠ Common exam trap
Google Cloud often tests the misconception that adjusting hyperparameters like temperature or top-k can fix factual accuracy and verbosity, when in reality these issues stem from the model's lack of external knowledge and lack of output constraints, which require architectural changes like RAG and constrained decoding.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Implement a retrieval-augmented generation (RAG) pipeline that queries a vector database of verified financial data, and apply constrained decoding with a maximum token limit and a repetition penalty.
It directly addresses both issues without retraining. A RAG pipeline grounds the model's outputs in verified financial data, eliminating factual inaccuracies. Constrained decoding with a maximum token limit and repetition penalty directly curbs verbosity and repetition, which temperature and top-k adjustments failed to fix.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Increase the LoRA rank to 32 and fine-tune the model for additional epochs on a curated subset of reports that focus on concise and accurate summaries.
Why it's wrong here
This would require additional training and may not guarantee reduction in verbosity; it also does not address real-time factual accuracy because fine-tuning on static data cannot correct errors from outdated or missing information.
- ✓
Implement a retrieval-augmented generation (RAG) pipeline that queries a vector database of verified financial data, and apply constrained decoding with a maximum token limit and a repetition penalty.
Why this is correct
This directly improves factual accuracy by grounding outputs in retrieved evidence and reduces verbosity through decoding constraints, without retraining.
- ✗
Switch to a larger pre-trained model (e.g., PaLM 2 or GPT-4) and use the same fine-tuning data with higher rank LoRA to improve capability, then rely on the larger model's inherent accuracy.
Why it's wrong here
Moving to a larger model is costly and time-consuming; it may still produce verbose output and factual errors without additional mechanisms like RAG.
- ✗
Experiment with higher temperature (e.g., 0.9) and lower top-k (e.g., 20) to encourage more diverse and concise outputs, and add a post-processing step to truncate summaries to 200 words.
Why it's wrong here
Higher temperature typically increases randomness and verbosity, contradicting the goal. Post-processing truncation may cut off important information and does not fix factual inaccuracies.
Go deeper
Related to this question
About these practice questions
One of 683 original Generative AI Leader practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This Generative AI Leader practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Generative AI Leader exam.