Scenario practice questions

A team is tuning a large language model for a question-answering task. They notice the model gives high confidence scores to answers that are factually incorrect. Which evaluation metric should they primarily use to detect this overconfidence problem?

Trap 1: Perplexity

Perplexity measures how fluently a model predicts token sequences, not whether stated confidence matches factual accuracy, so it cannot expose overconfidence. It is tempting because perplexity is a standard intrinsic metric for language modelling and detects distributional drift; it would suit comparing model versions or detecting unfamiliar inputs, not calibration of answer correctness.

Trap 2: BLEU score

BLEU compares n-gram overlap between generated and reference text, so a confidently wrong answer sharing wording with a reference scores well and the miscalibration stays hidden. It is tempting because BLEU is a long-standing translation metric; it would be the right choice when scoring surface similarity to reference translations, not for measuring confidence against factual correctness.

Trap 3: ROUGE-L

ROUGE-L measures longest-common-subsequence overlap with a reference, which rewards fluent phrasing regardless of factual truth, leaving overconfident errors undetected. It is tempting because ROUGE is standard for summarisation evaluation; it would be correct when judging how much reference content a summary recalls, not for assessing whether confidence scores are calibrated.

A
Perplexity
Why it fails: Perplexity measures how fluently a model predicts token sequences, not whether stated confidence matches factual accuracy, so it cannot expose overconfidence. It is tempting because perplexity is a standard intrinsic metric for language modelling and detects distributional drift; it would suit comparing model versions or detecting unfamiliar inputs, not calibration of answer correctness.
B
Expected Calibration Error (ECE)
Expected Calibration Error directly measures the gap between predicted confidence and actual accuracy across probability bins, so systematically high confidence on wrong answers produces a large ECE value. This satisfies the stem's overconfidence constraint, unlike accuracy or F1, which ignore confidence entirely and cannot expose miscalibration.
C
BLEU score
Why it fails: BLEU compares n-gram overlap between generated and reference text, so a confidently wrong answer sharing wording with a reference scores well and the miscalibration stays hidden. It is tempting because BLEU is a long-standing translation metric; it would be the right choice when scoring surface similarity to reference translations, not for measuring confidence against factual correctness.
D
ROUGE-L
Why it fails: ROUGE-L measures longest-common-subsequence overlap with a reference, which rewards fluent phrasing regardless of factual truth, leaving overconfident errors undetected. It is tempting because ROUGE is standard for summarisation evaluation; it would be correct when judging how much reference content a summary recalls, not for assessing whether confidence scores are calibrated.

Question 2mediummultiple choice

A company is evaluating whether to use a pre-built API or fine-tune a model for their use case. They have a large dataset of domain-specific jargon and need high accuracy on specialized terms. Which factor MOST strongly suggests fine-tuning?

Trap 1: The team wants to rapidly prototype a solution

Pre-built APIs are faster for prototyping.

Trap 2: The team has a limited budget for compute resources

A limited compute budget argues against fine-tuning, since training and hosting a tuned model consumes GPU capacity and cost. Fine-tuning is justified by domain-specific jargon and accuracy needs, which the stem already supplies. Budget constraints would instead favour a pre-built API with prompt engineering.

Trap 3: The application requires low latency responses

Low latency favours a pre-built API, because fine-tuned models add inference overhead and hosting complexity without improving response time. Fine-tuning addresses specialised terminology accuracy, not speed. Latency requirements would be the deciding factor when choosing between hosted endpoints or smaller distilled models.

A
The team wants to rapidly prototype a solution
Why it fails: Pre-built APIs are faster for prototyping.
B
The team has a limited budget for compute resources
Why it fails: A limited compute budget argues against fine-tuning, since training and hosting a tuned model consumes GPU capacity and cost. Fine-tuning is justified by domain-specific jargon and accuracy needs, which the stem already supplies. Budget constraints would instead favour a pre-built API with prompt engineering.
C
The application requires low latency responses
Why it fails: Low latency favours a pre-built API, because fine-tuned models add inference overhead and hosting complexity without improving response time. Fine-tuning addresses specialised terminology accuracy, not speed. Latency requirements would be the deciding factor when choosing between hosted endpoints or smaller distilled models.
D
The model needs to understand and generate domain-specific jargon accurately
Fine-tuning updates a model's weights on domain-specific text, embedding specialised jargon and terminology into its parameters. A pre-built API cannot reliably interpret or generate such vocabulary, so the large jargon dataset and high-accuracy requirement on specialised terms are the constraints that make fine-tuning the appropriate choice.

Question 3mediummulti select

A fintech company is deploying a generative AI system that offers investment advice. To comply with regulations and Google's AI Principles, they need to ensure appropriate human oversight and transparency. Which two actions should they take? (Choose two.)

Trap 1: Use only synthetic data for training to avoid privacy issues

Synthetic data addresses privacy and training-data scarcity, not human oversight or transparency in advice delivery. It is tempting because synthetic data genuinely supports privacy compliance, and would be correct if the requirement were avoiding personal data exposure rather than ensuring oversight.

Trap 2: Allow the AI to execute trades automatically without human…

Automatic trade execution removes the human oversight the scenario requires, and opaque automated decisions breach transparency obligations. It is tempting because straight-through processing suits low-risk, high-volume execution, but would only be acceptable where humans retain review authority over advice and trades.

Trap 3: Disable logging of all interactions to maximize user privacy

Disabling logging removes the audit trail that regulators and Google's AI Principles require for tracing advisory outputs back to their inputs, defeating transparency and oversight. Logging is tempting when privacy is the goal, and it would suit a system handling no regulated decisions or accountability requirements.

A
Use only synthetic data for training to avoid privacy issues
Why it fails: Synthetic data addresses privacy and training-data scarcity, not human oversight or transparency in advice delivery. It is tempting because synthetic data genuinely supports privacy compliance, and would be correct if the requirement were avoiding personal data exposure rather than ensuring oversight.
B
Allow the AI to execute trades automatically without human intervention
Why it fails: Automatic trade execution removes the human oversight the scenario requires, and opaque automated decisions breach transparency obligations. It is tempting because straight-through processing suits low-risk, high-volume execution, but would only be acceptable where humans retain review authority over advice and trades.
C
Require a human advisor to review and approve all AI-generated recommendations
Human-in-the-loop review places a qualified adviser between the model's output and the client, catching erroneous or unsuitable recommendations before they take effect. This delivers the human oversight the stem demands for regulated investment advice, aligning with Google's AI Principles on accountability and responsible deployment.
D
Develop and publish a Model Card that describes the model's limitations and intended use
A Model Card documents intended use, limitations, performance characteristics and ethical considerations, giving regulators, advisers and clients the transparency the stem requires. Publishing it satisfies Google's AI Principles on being accountable and understandable, complementing human review of AI-generated investment recommendations.
E
Disable logging of all interactions to maximize user privacy
Why it fails: Disabling logging removes the audit trail that regulators and Google's AI Principles require for tracing advisory outputs back to their inputs, defeating transparency and oversight. Logging is tempting when privacy is the goal, and it would suit a system handling no regulated decisions or accountability requirements.

Question 4mediummultiple choice

A media company wants its editorial staff to draft blog posts inside a web-based workspace where Gemini can summarize uploaded research PDFs, generate outlines, and cite files from the team's shared drive, all without writing code or managing any Google Cloud infrastructure. Which Google Cloud generative AI offering best fits this requirement?

Trap 1: Google Cloud Vertex AI Agent Builder with a custom tool

Agent Builder is aimed at developers building conversational agents and search applications that call tools and data stores. Standing it up requires design, configuration, and deployment work, and it delivers an agent endpoint rather than an in-app drafting experience for editors, so it does not satisfy the no-code workspace and zero-infrastructure constraints.

Trap 2: Gemini Enterprise with a connected data store

Gemini Enterprise targets organizations that want a standalone, enterprise-grade AI assistant with governance and connectors across business systems. While it is a managed product, it is a separate assistant surface rather than Gemini embedded in Docs and Drive, so it does not meet the requirement that editors work inside their existing document workspace.

Trap 3: Vertex AI Studio with a tuned Gemini model endpoint

Vertex AI Studio is a developer-oriented console for prompt design, tuning, and evaluation, and it produces endpoints that applications call programmatically. It does not deliver an end-user productivity workspace with native document grounding on a shared drive for non-technical editors, so it fails the stated no-code, out-of-the-box workspace requirement.

A
Google Cloud Vertex AI Agent Builder with a custom tool
Why it fails: Agent Builder is aimed at developers building conversational agents and search applications that call tools and data stores. Standing it up requires design, configuration, and deployment work, and it delivers an agent endpoint rather than an in-app drafting experience for editors, so it does not satisfy the no-code workspace and zero-infrastructure constraints.
B
Gemini for Google Workspace
Gemini for Google Workspace embeds Gemini directly in Gmail, Docs, Drive, and related apps, letting non-technical staff summarize PDFs, draft content, and ground responses in files they can already access. It requires no infrastructure work, which matches the editorial team's need for a no-code workspace experience with shared-drive content available in context.
C
Gemini Enterprise with a connected data store
Why it fails: Gemini Enterprise targets organizations that want a standalone, enterprise-grade AI assistant with governance and connectors across business systems. While it is a managed product, it is a separate assistant surface rather than Gemini embedded in Docs and Drive, so it does not meet the requirement that editors work inside their existing document workspace.
D
Vertex AI Studio with a tuned Gemini model endpoint
Why it fails: Vertex AI Studio is a developer-oriented console for prompt design, tuning, and evaluation, and it produces endpoints that applications call programmatically. It does not deliver an end-user productivity workspace with native document grounding on a shared drive for non-technical editors, so it fails the stated no-code, out-of-the-box workspace requirement.

Question 5mediummultiple choice

A company fine-tunes a model using Vertex AI and notices the model's performance drops on the original training task (e.g., language understanding) after fine-tuning for a new task (e.g., summarization). What could be the cause?

Trap 1: Data leakage

Data leakage inflates evaluation metrics; it does not degrade the original task. It tempts because leakage is genuinely the cause when validation scores look implausibly high, whereas the described drop on the original task points to catastrophic forgetting during fine-tuning.

Trap 2: Model quantization

Quantization reduces numerical precision, costing some accuracy, but it does not selectively erase the original task while a new one is learned. It tempts because quantization is genuinely the cause when a deployed model loses accuracy after compression for latency or memory savings.

Trap 3: Underfitting

Underfitting means the model fails to learn the training data at all, producing poor results everywhere, not a drop on the original task after new training. It is tempting because both involve weak performance, but underfitting is diagnosed from high training loss, whereas this scenario describes catastrophic forgetting during fine-tuning.

A
Data leakage
Why it fails: Data leakage inflates evaluation metrics; it does not degrade the original task. It tempts because leakage is genuinely the cause when validation scores look implausibly high, whereas the described drop on the original task points to catastrophic forgetting during fine-tuning.
B
Model quantization
Why it fails: Quantization reduces numerical precision, costing some accuracy, but it does not selectively erase the original task while a new one is learned. It tempts because quantization is genuinely the cause when a deployed model loses accuracy after compression for latency or memory savings.
C
Catastrophic forgetting
Catastrophic forgetting occurs when gradient updates for the new summarisation task overwrite the weights encoding the original language-understanding capability. This directly explains the performance drop on the earlier task, satisfying the stem's observation of degraded prior-task accuracy after fine-tuning.
D
Underfitting
Why it fails: Underfitting means the model fails to learn the training data at all, producing poor results everywhere, not a drop on the original task after new training. It is tempting because both involve weak performance, but underfitting is diagnosed from high training loss, whereas this scenario describes catastrophic forgetting during fine-tuning.

Question 6hardmultiple choice

A company is migrating a GenAI proof-of-concept to production. During the pilot, they used a large model (e.g., Gemini 1.5 Pro) and incurred high costs. The use case is simple: generating short product descriptions from structured data. Which cost optimization strategy should they implement first?

Trap 1: Fine-tune a smaller model on the specific task

Fine-tuning a smaller model would require curating a high-quality labelled dataset of product descriptions and retraining, which introduces latency and maintenance overhead for a simple task that could be solved with prompt engineering on a cheaper, smaller model. It is tempting because fine-tuning is the standard approach for adapting a model to a domain-specific task when the base model lacks the required knowledge, and would be correct if the structured data contained proprietary terminology or formatting not handled by zero-shot prompting.

Trap 2: Implement batch processing to group requests

Batch processing groups asynchronous requests, which suits non-interactive bulk workloads; short product descriptions generated on demand need immediate responses, so batching adds latency without addressing the cost driver. The primary saving comes from matching model size to task complexity, since a smaller model handles this simple structured-input task.

Trap 3: Reduce the model's temperature to 0.0

Temperature controls sampling randomness, not token pricing; setting it to 0.0 makes output deterministic but leaves per-token cost unchanged. It would matter when consistent, repeatable responses are required, yet the cost problem stems from using an oversized model for a simple task, which a smaller model resolves.

A
Fine-tune a smaller model on the specific task
Why it fails: Fine-tuning a smaller model would require curating a high-quality labelled dataset of product descriptions and retraining, which introduces latency and maintenance overhead for a simple task that could be solved with prompt engineering on a cheaper, smaller model. It is tempting because fine-tuning is the standard approach for adapting a model to a domain-specific task when the base model lacks the required knowledge, and would be correct if the structured data contained proprietary terminology or formatting not handled by zero-shot prompting.
B
Implement batch processing to group requests
Why it fails: Batch processing groups asynchronous requests, which suits non-interactive bulk workloads; short product descriptions generated on demand need immediate responses, so batching adds latency without addressing the cost driver. The primary saving comes from matching model size to task complexity, since a smaller model handles this simple structured-input task.
C
Reduce the model's temperature to 0.0
Why it fails: Temperature controls sampling randomness, not token pricing; setting it to 0.0 makes output deterministic but leaves per-token cost unchanged. It would matter when consistent, repeatable responses are required, yet the cost problem stems from using an oversized model for a simple task, which a smaller model resolves.
D
Switch to a smaller model like Gemini 1.5 Flash and use structured prompts
Gemini 1.5 Flash handles short structured generation at far lower cost per token than Pro, and structured prompts reduce token overhead further. Since the task is simple, this preserves output quality while directly addressing the high-cost constraint identified during the pilot.

Question 7mediummulti select

A company wants to deploy a GenAI code review assistant that integrates into their existing Git workflow. They want to use a managed Google Cloud service to minimize operational overhead. Which TWO services should they consider? (Choose two.)

Trap 1: Compute Engine

Compute Engine provides raw IaaS virtual machines, leaving patching, scaling and runtime management to the customer, which contradicts the minimal-operational-overhead requirement. It is tempting because it hosts arbitrary workloads, and would suit lift-and-shift migration of a legacy application needing full OS control.

Trap 2: Cloud Spanner

Cloud Spanner is a globally distributed relational database, so it cannot host or serve the code review assistant's model or application logic. It is tempting because it is fully managed, and would be the right choice for a transactional application requiring horizontal scale and strong consistency across regions.

Trap 3: Cloud Run

Cloud Run runs containerised request-serving workloads, but the scenario needs a managed service that ingests Git repositories and applies generative models to code. It is tempting because it is serverless and low-overhead, and would suit deploying a custom containerised API when no purpose-built managed service fits.

A
Cloud Build
Cloud Build is a managed CI/CD service that natively triggers builds from Git repositories, satisfying the Git-workflow integration and low operational overhead constraints. It runs the review pipeline serverlessly, so the team avoids provisioning or patching build infrastructure themselves.
B
Compute Engine
Why it fails: Compute Engine provides raw IaaS virtual machines, leaving patching, scaling and runtime management to the customer, which contradicts the minimal-operational-overhead requirement. It is tempting because it hosts arbitrary workloads, and would suit lift-and-shift migration of a legacy application needing full OS control.
C
Cloud Spanner
Why it fails: Cloud Spanner is a globally distributed relational database, so it cannot host or serve the code review assistant's model or application logic. It is tempting because it is fully managed, and would be the right choice for a transactional application requiring horizontal scale and strong consistency across regions.
D
Vertex AI Agent Builder
Vertex AI Agent Builder provides a managed environment for building and orchestrating GenAI agents, satisfying the minimal-operational-overhead constraint. It supplies the reasoning and tool-calling layer that inspects code changes and posts review comments, without the team managing underlying model infrastructure.
E
Cloud Run
Why it fails: Cloud Run runs containerised request-serving workloads, but the scenario needs a managed service that ingests Git repositories and applies generative models to code. It is tempting because it is serverless and low-overhead, and would suit deploying a custom containerised API when no purpose-built managed service fits.

Question 8hardmulti select

A global e-commerce company uses generative AI to generate product descriptions in multiple languages. They want to ensure consistency across markets while respecting cultural nuances. Which THREE strategies should they adopt?

Trap 1: Standardize all descriptions to a neutral tone to avoid cultural…

A single neutral tone strips the local idiom, humour and cultural references that make descriptions resonate, undermining the nuance requirement. Neutral standardisation suits regulated, safety-critical or legally sensitive content where uniform wording outweighs market-localised engagement.

Trap 2: Use a single global model with a translation layer to convert…

A translation layer converts surface text but cannot regenerate culturally native phrasing, idioms or market-specific connotations, so nuance is lost. A single global model with translation suits internal documentation or low-stakes content where literal accuracy matters more than local resonance.

A
Standardize all descriptions to a neutral tone to avoid cultural issues.
Why it fails: A single neutral tone strips the local idiom, humour and cultural references that make descriptions resonate, undermining the nuance requirement. Neutral standardisation suits regulated, safety-critical or legally sensitive content where uniform wording outweighs market-localised engagement.
B
Develop region-specific prompt templates that incorporate local cultural references and legal requirements.
Region-specific prompt templates embed local cultural references and legal requirements into generation, ensuring each market's descriptions respect nuances while a shared template structure maintains cross-market consistency. This directly satisfies both the consistency and cultural-respect constraints in the stem.
C
Engage local marketing teams to review and approve AI-generated descriptions before publication.
Local marketing teams provide the cultural nuance that a single global prompt cannot encode, catching idioms, taboos and tone that literal translation misses. Their review-and-approval step enforces consistency across markets while adapting each description locally, directly satisfying the stem's dual requirement of uniformity and cultural sensitivity.
D
Use a single global model with a translation layer to convert English descriptions.
Why it fails: A translation layer converts surface text but cannot regenerate culturally native phrasing, idioms or market-specific connotations, so nuance is lost. A single global model with translation suits internal documentation or low-stakes content where literal accuracy matters more than local resonance.
E
Use A/B testing to measure engagement metrics per region and iterate on prompts.
A/B testing per region supplies the empirical feedback loop the stem demands: engagement metrics reveal whether a prompt's tone or idiom lands culturally, letting the team iterate prompts rather than assume one wording suits every market. This directly satisfies the consistency-with-cultural-nuance constraint.

Question 9mediummulti select

A company is deploying a generative AI system for medical diagnosis. Which TWO measures are essential for responsible AI in this high-stakes domain?

Trap 1: Allow the AI to make autonomous decisions in time-sensitive…

Autonomous AI decisions in emergencies remove the clinician oversight that medical diagnosis demands, so accountability for harmful errors cannot be assigned. It is tempting because faster triage genuinely saves lives, but autonomous emergency decisioning belongs in low-stakes, reversible contexts, not diagnosis.

Trap 2: Publish all patient data used for training to ensure transparency

Publishing patient data breaches confidentiality and privacy law, and transparency about model behaviour does not require exposing identifiable records. It is tempting because transparency is a genuine responsible-AI principle, but it is satisfied through model cards, documentation and audit processes, not raw training data release.

Trap 3: Use the AI only for administrative tasks, not diagnosis

Restricting the AI to administrative tasks abandons the diagnostic use case entirely, so it delivers no responsible-AI measure for the system being deployed. It is tempting because avoiding diagnosis sidesteps clinical risk, but the question asks how to govern diagnostic AI, not how to avoid it.

A
Allow the AI to make autonomous decisions in time-sensitive emergencies
Why it fails: Autonomous AI decisions in emergencies remove the clinician oversight that medical diagnosis demands, so accountability for harmful errors cannot be assigned. It is tempting because faster triage genuinely saves lives, but autonomous emergency decisioning belongs in low-stakes, reversible contexts, not diagnosis.
B
Ensure a human medical professional reviews all AI-generated diagnoses
Human-in-the-loop review directly addresses the clinical risk constraint: an erroneous AI diagnosis could cause patient harm. Because generative models can hallucinate confidently, a qualified clinician must validate every output before it informs care, preserving professional accountability and satisfying medical-device oversight expectations for high-stakes decisions.
C
Publish all patient data used for training to ensure transparency
Why it fails: Publishing patient data breaches confidentiality and privacy law, and transparency about model behaviour does not require exposing identifiable records. It is tempting because transparency is a genuine responsible-AI principle, but it is satisfied through model cards, documentation and audit processes, not raw training data release.
D
Provide model documentation (Model Cards) to clinicians detailing the system's limitations
Model Cards expose a model's intended use, training data provenance, performance disparities and known failure modes, so clinicians can judge when outputs are untrustworthy. In medical diagnosis, where erroneous recommendations cause direct patient harm, this transparency satisfies the responsible-AI requirement for accountability and informed human oversight before acting on generated advice.
E
Use the AI only for administrative tasks, not diagnosis
Why it fails: Restricting the AI to administrative tasks abandons the diagnostic use case entirely, so it delivers no responsible-AI measure for the system being deployed. It is tempting because avoiding diagnosis sidesteps clinical risk, but the question asks how to govern diagnostic AI, not how to avoid it.

Question 10hardmultiple choice

A retail company is building a generative AI chatbot to assist customers with product recommendations and order tracking. The chatbot uses Vertex AI with Gemini 1.5 Pro, and the development team has implemented a Retrieval-Augmented Generation (RAG) pipeline using Vertex AI Search for grounding. The pipeline uses a vector store containing product descriptions and order history. During testing, the team observes that the chatbot sometimes provides incorrect order statuses—for example, claiming an order is 'shipped' when it is actually 'pending'. The team suspects the issue is related to how context is retrieved and used. The RAG pipeline currently retrieves the top 5 chunks based on cosine similarity from the vector store, and passes them as context to the model. The team is considering several changes to improve factual accuracy. Which single action would most effectively reduce hallucinations in this scenario?

Trap 1: Switch from Vertex AI Search to a different vector database like…

Switching to a different vector database like Pinecone does not address the core issue of low-relevance retrieval; the quality of similarity search depends on the embedding and threshold, not the database vendor.

Trap 2: Reduce the model temperature to 0.0 to make outputs more…

Reducing temperature to 0.0 increases determinism but does not fix hallucinations caused by irrelevant or incorrect context; the model may still confidently generate wrong answers based on poor retrieval.

Trap 3: Increase the top-K retrieval value to 10 to provide more context to…

Increasing top-K to 10 adds more chunks, but if those additional chunks have low similarity, they introduce irrelevant or conflicting information, potentially worsening hallucinations instead of improving accuracy.

A
Switch from Vertex AI Search to a different vector database like Pinecone.
Why it fails: Switching to a different vector database like Pinecone does not address the core issue of low-relevance retrieval; the quality of similarity search depends on the embedding and threshold, not the database vendor.
B
Reduce the model temperature to 0.0 to make outputs more deterministic.
Why it fails: Reducing temperature to 0.0 increases determinism but does not fix hallucinations caused by irrelevant or incorrect context; the model may still confidently generate wrong answers based on poor retrieval.
C
Increase the similarity score threshold for retrieval to 0.85 to filter out less relevant chunks.
Increasing the similarity threshold to 0.85 filters out low-relevance chunks (e.g., order statuses from other customers), ensuring the model receives only highly relevant context, which directly reduces hallucinated order statuses in this RAG pipeline.
D
Increase the top-K retrieval value to 10 to provide more context to the model.
Why it fails: Increasing top-K to 10 adds more chunks, but if those additional chunks have low similarity, they introduce irrelevant or conflicting information, potentially worsening hallucinations instead of improving accuracy.