The correct action is to periodically clear the key-value cache between inference requests and implement cache truncation for long sequences. This resolves the latency spike because the unbounded growth of the KV cache across requests directly causes the observed 300% increase in inference latency—the cache consumes excessive GPU memory, forcing high data transfer wait times as the system swaps data between memory and compute, leaving GPUs idle. On the Google Cloud Generative AI Leader exam, this scenario tests your understanding of LLM inference optimization, specifically how KV cache management prevents memory bloat in production deployments on Vertex AI. A common trap is to assume the issue requires hardware upgrades or model recompilation, but the root cause is simply the cache persisting across requests. Memory tip: think of the KV cache like a growing backlog—clear it between calls to keep inference fast and lean.
Generative AI Leader Fundamentals of Generative AI Practice Question
This Generative AI Leader practice question tests your understanding of fundamentals of generative ai. The scenario asks you to isolate a root cause — eliminate options that address a different problem before choosing. After answering, compare your reasoning against the explanation and wrong-answer breakdown below. Once you have made your selection, read the full explanation to reinforce the concept and understand why each distractor is designed to mislead on exam day.
You are the lead AI engineer at a financial services firm. You have fine-tuned a large language model on historical trade reports to generate daily market summaries. The model is deployed on Google Cloud's Vertex AI using a custom container. A few weeks after deployment, the operations team notices that inference latency has increased by 300%, causing timeouts. You investigate and find that the model's memory consumption has grown unexpectedly, and the GPUs are idling due to high data transfer wait times. The model architecture and code have not changed. Which action is most likely to resolve the latency issue?
Clue words in this question
Noticing these words before you look at the options changes how you read each choice.
Clue: "most likely"
Why it matters: Probability qualifier — the question wants the most probable cause or outcome, not a guaranteed one. Eliminate low-probability options.
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Periodically clear the key-value cache between inference requests and implement cache truncation for long sequences.
The latency spike is caused by the key-value (KV) cache growing unboundedly across inference requests, leading to excessive memory consumption and data transfer wait times. Periodically clearing the KV cache between requests and truncating it for long sequences directly addresses the root cause by freeing GPU memory and reducing I/O bottlenecks, without requiring hardware upgrades or recompilation.
Key principle: Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
✗
Upgrade to a more powerful GPU instance (e.g., A100 to H100) to handle the increased memory footprint.
Why it's wrong here
The issue is memory fragmentation, not insufficient GPU compute.
✗
Enable preemptible VM instances to reduce cost and redeploy the model on a faster network.
Why it's wrong here
Preemptible instances are not more reliable and don't solve memory issues.
✓
Periodically clear the key-value cache between inference requests and implement cache truncation for long sequences.
Why this is correct
Clearing and managing the KV cache reduces memory bloat and speeds up inference.
Clue confirmation
The clue word "most likely" in the question point toward this answer.
Related concept
Read the scenario before looking for a memorised answer.
✗
Recompile the model using XLA with optimizations for dynamic shapes.
Why it's wrong here
Recompilation might not fix runtime memory fragmentation.
Common exam traps
Common exam trap: answer the scenario, not the keyword
Google Cloud often tests the misconception that hardware upgrades or compilation optimizations can fix memory management issues, when the real problem is a software-level cache leak that must be handled explicitly in the serving infrastructure.
Detailed technical explanation
How to think about this question
In transformer-based LLMs, the KV cache stores previous token keys and values to avoid recomputation during autoregressive decoding. If not cleared between requests (e.g., due to a shared serving session or improper batching logic), the cache grows linearly with total tokens processed, causing memory fragmentation and increased PCIe transfers that starve GPU compute. Real-world deployments often implement cache eviction policies (e.g., sliding window or attention sink) to bound memory usage while maintaining generation quality.
KKey Concepts to Remember
Read the scenario before looking for a memorised answer.
Find the constraint that changes the correct option.
Eliminate answers that are true in general but not in this case.
TExam Day Tips
→Watch for words such as best, first, most likely and least administrative effort.
→Review why wrong options are wrong, not only why the correct option is correct.
Key takeaway
Answer the scenario, not the keyword: identify the specific constraint before choosing the most familiar-sounding option.
Real-world example
How this comes up in practice
A startup's cloud architect reviews their monthly bill and notices costs are higher than expected for a long-running batch job. Switching from on-demand instances to Reserved Instances — or using Spot/Preemptible VMs — can reduce compute costs by up to 72 %. Questions like this test whether you understand the tradeoffs between commitment, flexibility, and cost across cloud pricing models.
What to study next
Got this wrong? Here's your next step.
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
What does this Generative AI Leader question test?
Fundamentals of Generative AI — This question tests Fundamentals of Generative AI — Read the scenario before looking for a memorised answer..
What is the correct answer to this question?
The correct answer is: Periodically clear the key-value cache between inference requests and implement cache truncation for long sequences. — The latency spike is caused by the key-value (KV) cache growing unboundedly across inference requests, leading to excessive memory consumption and data transfer wait times. Periodically clearing the KV cache between requests and truncating it for long sequences directly addresses the root cause by freeing GPU memory and reducing I/O bottlenecks, without requiring hardware upgrades or recompilation.
What should I do if I get this Generative AI Leader question wrong?
Identify which exam domain this question belongs to, review the core concept, then practise similar questions from the same domain.
Are there clue words in this question I should notice?
Yes — watch for: "most likely". Probability qualifier — the question wants the most probable cause or outcome, not a guaranteed one. Eliminate low-probability options.
What is the key concept behind this question?
Read the scenario before looking for a memorised answer.
About these practice questions
Courseiva creates original exam-style practice questions with explanations and wrong-answer analysis. It does not publish real exam questions, exam dumps, or protected exam content. Learn why practice questions differ from exam dumps →
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
This Generative AI Leader practice question is part of Courseiva's free Google Cloud certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the Generative AI Leader exam.
Question Discussion
Share a tip, memory trick, or ask about the reasoning behind this question. Do not post real exam questions, leaked content, braindumps, or copyrighted exam material. Comments are moderated and may be removed without notice.
Sign in to join the discussion.