Scenario practice questions

A financial institution is deploying a generative AI solution that generates investment advice. They must ensure fairness, avoid toxic outputs, and comply with regulations like GDPR. Which TWO strategies should they implement? (Choose two.)

Trap 1: Set the model temperature to 0 to eliminate creativity and reduce…

E is wrong because temperature 0 does not eliminate bias; it only makes outputs deterministic.

Trap 2: Fine-tune the model exclusively on compliant financial documents.

C is wrong because fine-tuning does not guarantee non-toxic outputs.

Trap 3: Disable request logging to avoid storing sensitive data.

A is wrong because GDPR often requires logging for audit.

A
Use Vertex AI Safety Attributes to filter harmful content in both input and output.
B is correct because it proactively blocks toxic content.
B
Set the model temperature to 0 to eliminate creativity and reduce bias.
Why wrong: E is wrong because temperature 0 does not eliminate bias; it only makes outputs deterministic.
C
Implement a human review process for any advice above a certain risk threshold.
D is correct because human oversight catches errors the model might miss.
D
Fine-tune the model exclusively on compliant financial documents.
Why wrong: C is wrong because fine-tuning does not guarantee non-toxic outputs.
E
Disable request logging to avoid storing sensitive data.
Why wrong: A is wrong because GDPR often requires logging for audit.

Question 2mediummultiple choice

A company fine-tunes a model using Vertex AI and notices the model's performance drops on the original training task (e.g., language understanding) after fine-tuning for a new task (e.g., summarization). What could be the cause?

Trap 1: Data leakage

Data leakage might cause overoptimistic evaluation, not a distinct performance drop on the original task.

Trap 2: Model quantization

Quantization reduces model size but typically does not cause forgetting of original tasks.

Trap 3: Underfitting

Underfitting would show poor performance on both tasks, not a drop on the original task alone.

A
Data leakage
Why wrong: Data leakage might cause overoptimistic evaluation, not a distinct performance drop on the original task.
B
Model quantization
Why wrong: Quantization reduces model size but typically does not cause forgetting of original tasks.
C
Catastrophic forgetting
Fine-tuning on a narrow task can overwrite general knowledge, leading to performance degradation on the original task.
D
Underfitting
Why wrong: Underfitting would show poor performance on both tasks, not a drop on the original task alone.

Question 3mediummultiple choice

A team is tuning a large language model for a question-answering task. They notice the model gives high confidence scores to answers that are factually incorrect. Which evaluation metric should they primarily use to detect this overconfidence problem?

Trap 1: Perplexity

Perplexity indicates model surprise, not confidence in correctness.

Trap 2: BLEU score

BLEU evaluates text similarity, not confidence calibration.

Trap 3: ROUGE-L

ROUGE-L measures recall of longest common subsequence, not calibration.

A
Perplexity
Why wrong: Perplexity indicates model surprise, not confidence in correctness.
B
Expected Calibration Error (ECE)
ECE directly quantifies how well confidence scores reflect actual correctness.
C
BLEU score
Why wrong: BLEU evaluates text similarity, not confidence calibration.
D
ROUGE-L
Why wrong: ROUGE-L measures recall of longest common subsequence, not calibration.

Read the full NAT/PAT explanation →

Question 4hardmultiple choice

A retail company is building a generative AI chatbot to assist customers with product recommendations and order tracking. The chatbot uses Vertex AI with Gemini 1.5 Pro, and the development team has implemented a Retrieval-Augmented Generation (RAG) pipeline using Vertex AI Search for grounding. The pipeline uses a vector store containing product descriptions and order history. During testing, the team observes that the chatbot sometimes provides incorrect order statuses—for example, claiming an order is 'shipped' when it is actually 'pending'. The team suspects the issue is related to how context is retrieved and used. The RAG pipeline currently retrieves the top 5 chunks based on cosine similarity from the vector store, and passes them as context to the model. The team is considering several changes to improve factual accuracy. Which single action would most effectively reduce hallucinations in this scenario?

Trap 1: Switch from Vertex AI Search to a different vector database like…

Option D is wrong because changing databases does not address retrieval relevance or hallucination directly.

Trap 2: Reduce the model temperature to 0.0 to make outputs more…

Option C is wrong because temperature controls randomness, not factual grounding from retrieved context.

Trap 3: Increase the top-K retrieval value to 10 to provide more context to…

Option B is wrong because more context can introduce noise and increase hallucination risk.

A
Switch from Vertex AI Search to a different vector database like Pinecone.
Why wrong: Option D is wrong because changing databases does not address retrieval relevance or hallucination directly.
B
Reduce the model temperature to 0.0 to make outputs more deterministic.
Why wrong: Option C is wrong because temperature controls randomness, not factual grounding from retrieved context.
C
Increase the similarity score threshold for retrieval to 0.85 to filter out less relevant chunks.
Option A is correct because a higher threshold reduces irrelevant context, directly improving factual grounding.
D
Increase the top-K retrieval value to 10 to provide more context to the model.
Why wrong: Option B is wrong because more context can introduce noise and increase hallucination risk.

Question 5mediummultiple choice

During model evaluation, a team observes good performance on training data but poor on validation data. Which regularization technique is most appropriate to address this?

Trap 1: Add more training data

More data helps reduce overfitting but is not a regularization technique itself.

Trap 2: Increase the learning rate

Higher learning rate can cause divergence, not reduce overfitting.

Trap 3: Use a larger batch size

Larger batch sizes can lead to sharper minima and may worsen generalization.

A
Add more training data
Why wrong: More data helps reduce overfitting but is not a regularization technique itself.
B
Increase the learning rate
Why wrong: Higher learning rate can cause divergence, not reduce overfitting.
C
Apply dropout
Dropout is a regularization method that prevents co-adaptation of neurons, reducing overfitting.
D
Use a larger batch size
Why wrong: Larger batch sizes can lead to sharper minima and may worsen generalization.

Question 6hardmultiple choice

An AI team is building a customer support chatbot for a telecom company using a fine-tuned LLM on Vertex AI. The model performs well on common issues but fails to answer correctly for rare or novel problems, often providing plausible-sounding but incorrect solutions. The team has a large corpus of internal troubleshooting documents. They want to minimize incorrect answers while keeping latency low. Which approach should they take?

Trap 1: Switch to a larger base model (e.g., Gemini Ultra) without any…

A larger model may still hallucinate on niche topics and increases latency.

Trap 2: Collect more data on rare issues and continue fine-tuning the model…

Fine-tuning requires large amounts of data and may not scale to every rare issue; also time-consuming.

Trap 3: Use a few-shot prompt with 10 examples of rare problems and…

Few-shot examples are limited and cannot cover the diversity of rare issues.

A
Switch to a larger base model (e.g., Gemini Ultra) without any retrieval.
Why wrong: A larger model may still hallucinate on niche topics and increases latency.
B
Implement a retrieval-augmented generation (RAG) pipeline using Vertex AI Search to fetch relevant documents before generating answers.
RAG dynamically retrieves relevant context, enabling accurate answers for rare issues.
C
Collect more data on rare issues and continue fine-tuning the model weekly.
Why wrong: Fine-tuning requires large amounts of data and may not scale to every rare issue; also time-consuming.
D
Use a few-shot prompt with 10 examples of rare problems and solutions.
Why wrong: Few-shot examples are limited and cannot cover the diversity of rare issues.

Read the full NAT/PAT explanation →

Question 7mediummultiple choice

A retail company is deploying a generative AI chatbot on Vertex AI to provide product recommendations. The chatbot uses a base foundation model with no fine-tuning. Users report that the chatbot sometimes gives offensive or insensitive responses. The team must quickly implement safety controls without modifying the model. They also want to reduce irrelevant off-topic answers. Which combination of techniques should they apply?

Trap 1: Fine-tune the model on a curated dataset of safe retail…

Fine-tuning is not allowed as per the constraint of no model modification.

Trap 2: Set temperature to 0.0 and top_p to 0.1.

Lowering temperature reduces randomness but does not filter toxic content.

Trap 3: Provide 50 few-shot examples of safe interactions.

Few-shot examples help but are not a robust safety filter; edge cases may still appear.

A
Fine-tune the model on a curated dataset of safe retail conversations.
Why wrong: Fine-tuning is not allowed as per the constraint of no model modification.
B
Set temperature to 0.0 and top_p to 0.1.
Why wrong: Lowering temperature reduces randomness but does not filter toxic content.
C
Enable Vertex AI Safety Filters and craft system instructions defining appropriate behavior.
Safety filters block harmful output and system instructions guide the model's tone and relevance.
D
Provide 50 few-shot examples of safe interactions.
Why wrong: Few-shot examples help but are not a robust safety filter; edge cases may still appear.