Generative AI Leader Fundamentals of Generative AI Practice Test 8 — 15 Questions

Question 1

A team is tuning a large language model for a question-answering task. They notice the model gives high confidence scores to answers that are factually incorrect. Which evaluation metric should they primarily use to detect this overconfidence problem?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Expected Calibration Error (ECE). Expected Calibration Error (ECE) directly measures the alignment between a model's predicted confidence and its actual accuracy. In this scenario, high confidence on incorrect answers indicates miscalibration, and ECE quantifies this mismatch by binning predictions by confidence and computing the average absolute difference between accuracy and confidence per bin.

Answer

Perplexity

Answer

BLEU score

Answer

ROUGE-L

Question 2

A generative AI model is trained on a dataset containing biased text. The team wants to debias the model without significantly sacrificing performance on the original task. Which approach is most appropriate?

Accepted Answer

Curate a smaller, balanced dataset that is representative of fair outcomes and fine-tune the model using a combination of the original data and this dataset with a regularization penalty on bias metrics.. It directly addresses bias in the training data by combining the original dataset with a curated, balanced dataset and applying a regularization penalty on bias metrics. This approach allows the model to retain performance on the original task while explicitly penalizing biased representations during fine-tuning, which is a standard technique in fairness-aware machine learning. The regularization term acts as a constraint that guides the optimization away from biased decision boundaries without requiring full retraining or architectural changes.

Answer

Train an adversarial classifier to predict protected attributes from the model's hidden representations and minimize that prediction accuracy.

Answer

Filter the original training dataset to remove all sentences containing biased terms or stereotypes.

Answer

After training, apply a separate classifier on the model's output logits to adjust the final predictions for fairness.

Question 3

A multimodal generative AI system processes both image and text inputs to produce captions. During inference, the image encoder sometimes produces noisy or missing features. Which architectural design decision best handles such input degradation without retraining?

Accepted Answer

Introduce a gating mechanism that learns to weigh image features based on confidence scores from the encoder.. A gating mechanism dynamically adjusts the contribution of image features based on confidence scores from the encoder, allowing the model to gracefully handle noisy or missing features without retraining. This architectural design learns to suppress unreliable image inputs and rely more on text or other modalities, ensuring robust caption generation under input degradation.

Answer

Train a separate variational autoencoder to produce a clean latent representation from the noisy image.

Answer

Increase the image encoder’s capacity to better extract robust features.

Answer

Apply standard image preprocessing (e.g., denoising) to all inputs before feeding to the encoder.

Question 4

Which TWO of the following are key differences between generative AI and discriminative AI? (Choose two.)

Accepted Answer

Generative models can create new data samples, while discriminative models only assign labels to existing data.. Generative AI models learn the underlying distribution of the data, enabling them to generate new, realistic samples (e.g., images, text) from the learned distribution. In contrast, discriminative models learn decision boundaries to classify or label existing data without the ability to create new data instances. This fundamental difference in capability—creation versus discrimination—is a core distinction between the two paradigms.

Answer

Generative models require less training data than discriminative models.

Answer

Generative models cannot be used for supervised learning tasks like classification.

Answer

Discriminative models always outperform generative models on tasks like image classification.

Question 5

Which THREE of the following are common techniques to reduce harmful biases in generative AI models? (Choose three.)

Accepted Answer

Use reinforcement learning from human feedback (RLHF) with a reward model that penalizes biased or unfair outputs.. A is correct because RLHF uses a reward model trained on human preferences to score model outputs, and explicitly penalizing biased or unfair outputs during fine-tuning directly reduces harmful biases. This technique aligns the model's behavior with human values by optimizing against a learned reward signal that captures bias-related concerns.

Answer

Decrease the model's temperature parameter to make outputs more deterministic.

Answer

Conduct a legal review of all generated outputs before release.

Question 6

Which THREE of the following are key considerations when deploying a generative AI model in a production environment with strict latency requirements? (Choose three.)

Accepted Answer

Implement speculative decoding to generate candidate tokens with a smaller draft model and verify with the large model.. Speculative decoding uses a smaller, faster draft model to generate candidate tokens, which are then verified by the large model in parallel. This reduces the number of sequential autoregressive steps, significantly lowering latency while maintaining output quality.

Answer

Deploy the largest model variant available to ensure highest quality.

Answer

Increase the inference batch size to maximize GPU utilization.

Question 7

You are a generative AI lead at a healthcare startup developing a system to summarize patient medical records for quick review by doctors. The system uses a fine-tuned LLM. After deployment, doctors report that the summaries often miss critical details like medication dosages and allergy information. The current pipeline preprocesses patient records by extracting text from EHR, feeding it to the LLM, and outputting a summary. The team has limited time and budget. They cannot retrain the model because it is hosted as a managed API. Which action should you take to most effectively improve the summarization quality without changing the model?

Accepted Answer

Revise the prompt to explicitly ask for medication dosages and allergies, and format the input text by adding headings (e.g., '### Medications') to emphasize important sections.. Prompt engineering is the most effective and cost-efficient way to improve LLM output without retraining or changing the model. By explicitly instructing the model to include medication dosages and allergies, and by structuring the input with clear headings, you guide the model's attention to critical sections, directly addressing the missing details. This approach leverages the LLM's existing capabilities and requires no changes to the hosted API or additional infrastructure.

Answer

Increase the maximum output token limit to force the model to include more details.

Answer

Replace the LLM with a simpler extractive summarization model that selects sentences from the original document.

Answer

Implement a retrieval-augmented generation (RAG) system that pulls supplementary data from external drug databases.

Question 8

You are the lead AI engineer at a financial services firm. You have fine-tuned a large language model on historical trade reports to generate daily market summaries. The model is deployed on Google Cloud's Vertex AI using a custom container. A few weeks after deployment, the operations team notices that inference latency has increased by 300%, causing timeouts. You investigate and find that the model's memory consumption has grown unexpectedly, and the GPUs are idling due to high data transfer wait times. The model architecture and code have not changed. Which action is most likely to resolve the latency issue?

Accepted Answer

Periodically clear the key-value cache between inference requests and implement cache truncation for long sequences.. The latency spike is caused by the key-value (KV) cache growing unboundedly across inference requests, leading to excessive memory consumption and data transfer wait times. Periodically clearing the KV cache between requests and truncating it for long sequences directly addresses the root cause by freeing GPU memory and reducing I/O bottlenecks, without requiring hardware upgrades or recompilation.

Answer

Upgrade to a more powerful GPU instance (e.g., A100 to H100) to handle the increased memory footprint.

Answer

Enable preemptible VM instances to reduce cost and redeploy the model on a faster network.

Answer

Recompile the model using XLA with optimizations for dynamic shapes.

Question 9

You are a generative AI architect at a social media company. You are tasked with building a content moderation system that uses a generative model to flag toxic comments. The system must have very low false positive rates (i.e., not flag harmless comments) to avoid user backlash, but it must catch nearly all toxic comments. You have a large dataset of labeled toxic and non-toxic comments. You plan to use a pre-trained LLM and fine-tune it for classification. During experimentation, you notice that the model's recall for toxic comments is high (95%) but its precision is low (60%), leading to many false positives. You need to improve precision without substantially reducing recall. Which approach should you try first?

Accepted Answer

Tune the classification threshold on a held-out validation set to a higher value (e.g., require higher probability to classify as toxic).. Tuning the classification threshold to a higher value directly addresses the low precision (high false positive rate) by requiring a higher confidence level before labeling a comment as toxic. This reduces false positives while maintaining high recall, as the model's underlying learned representations remain unchanged. The threshold adjustment is a standard post-hoc calibration technique that trades off precision and recall without retraining or altering the model architecture.

Answer

Gather additional toxic comments from similar platforms to augment the training data.

Answer

Apply a higher weight to the toxic class in the loss function during fine-tuning.

Answer

Use a smaller pre-trained model that is inherently less sensitive to subtle toxic language.

Question 10

You are the AI lead at an e-commerce company that uses a generative model to write product descriptions from images and key attributes. The model is a multimodal transformer that encodes both image and text (attributes) and decodes a description. Recently, your team deployed a new version of the image encoder that uses a more powerful backbone (ViT-L instead of ViT-B). After deployment, the generated descriptions became longer but often include irrelevant visual details (e.g., background objects) and occasionally misrepresent the product's main features. The model was fine-tuned on the same dataset as before. The descriptions from the old model were concise and focused. What is the most likely cause of the degradation and the best fix?

Accepted Answer

The powerful encoder introduces overfitting to the training images; continue fine-tuning with additional loss terms that penalize description of irrelevant details (e.g., using attention regularization).. The more powerful ViT-L encoder captures richer, high-resolution features, including background details, which the decoder then amplifies into longer, less focused descriptions. This is a form of overfitting to irrelevant visual patterns in the training images, not to the labels. Adding attention regularization (e.g., penalizing attention weights on non-salient regions) forces the model to focus on product-relevant features, restoring conciseness and accuracy without reverting to the weaker encoder.

Answer

The decoder is now too small relative to the encoder; reduce the encoder's hidden size or increase the decoder's capacity.

Answer

The new encoder produces less discriminative features; replace it with an older version.

Answer

Lower the decoder's temperature to reduce diversity and hallucination.

Question 11

A company is building a conversational AI using the Gemini API on Vertex AI. They want to reduce the chance of generating toxic content while still allowing creative and engaging responses for their gaming community. Which TWO safety settings should they adjust in the safety_settings parameter?

Accepted Answer

Enable the 'harm_category' filter for 'DANGEROUS_CONTENT' with threshold BLOCK_ONLY_HIGH.. Setting the 'DANGEROUS_CONTENT' category to BLOCK_ONLY_HIGH allows the model to generate creative and engaging responses for a gaming community while still blocking the most severe dangerous content. This balances safety with creative freedom, as the gaming context may involve simulated conflict or action that is not genuinely harmful.

Answer

Set the threshold for 'SEXUALLY_EXPLICIT' category to BLOCK_NONE.

Answer

Set the threshold for 'SEXUALLY_EXPLICIT' category to BLOCK_LOW_AND_ABOVE.

Answer

Set the threshold for 'HARASSMENT' category to BLOCK_LOW_AND_ABOVE.

Question 12

A startup is developing a customer support chatbot using Vertex AI PaLM 2 API. They notice that the model sometimes generates plausible-sounding but factually incorrect information about company policies. The chatbot currently uses no external data. To reduce these hallucinations without retraining the model, the team needs a solution that can be implemented quickly and maintains low latency. They have access to the company's internal policy database stored in Cloud SQL. Which approach should they take?

Accepted Answer

Implement grounding by connecting the model to the company's policy database using Vertex AI Grounding.. Vertex AI Grounding connects the PaLM 2 model to the company's policy database in Cloud SQL, allowing the model to retrieve and cite factual information in real time. This approach reduces hallucinations without retraining, meets the low-latency requirement, and leverages existing internal data. Grounding works by augmenting the prompt with retrieved context from the grounding source, ensuring responses are factually grounded.

Answer

Fine-tune the PaLM 2 model on a dataset of company policy documents.

Answer

Reduce the temperature parameter to 0 and increase top_k to 50.

Answer

Use prompt engineering to instruct the model to only answer from its internal knowledge.

Question 13

A marketing team wants to generate product descriptions using a text generation model on Vertex AI. They need consistent output style across all descriptions, including tone and length. They have a small set of 10 high-quality example descriptions that capture the desired style. The team has limited ML expertise and wants a quick solution that does not require model retraining. Which approach should they use?

Accepted Answer

Use few-shot prompting with the examples in the prompt.. Few-shot prompting is the correct approach because it allows the team to inject the desired style, tone, and length directly into the prompt using the 10 high-quality examples, without any model retraining. This technique leverages the in-context learning capability of large language models on Vertex AI, enabling consistent output from a small set of demonstrations. It is ideal for teams with limited ML expertise as it requires only prompt engineering, not fine-tuning or infrastructure changes.

Answer

Use a pre-built template with no model input.

Answer

Fine-tune the model on a large external dataset of product descriptions.

Answer

Set the temperature to 0.9 to maximize creativity.

Question 14

A healthcare company is building a clinical decision support system using Gemini 1.5 Pro on Vertex AI. They need responses that are highly accurate and comply with medical regulations, including traceability to source documents. They have a large corpus of curated medical guidelines stored in PDFs in Cloud Storage. Their team has experience with both fine-tuning and prompt engineering. Which approach best ensures regulatory compliance and accuracy?

Accepted Answer

Use a combination of grounding to the medical guidelines and prompt engineering with system instructions specifying compliance requirements.. Grounding the model to the curated medical guidelines in Cloud Storage ensures responses are directly traceable to source documents, which is critical for medical regulatory compliance. Combining this with system instructions that specify compliance requirements (e.g., HIPAA, FDA guidelines) enforces behavioral constraints without altering the model's weights, maintaining accuracy and auditability.

Answer

Use prompt engineering with system instructions and few-shot examples, but no grounding.

Answer

Use grounding to the medical guidelines but rely on prompt engineering only for compliance instructions.

Answer

Fine-tune the model on the medical guidelines corpus to internalize the knowledge.

Question 15

A company is deploying a Gemini 1.0 Ultra model for a code generation assistant. They have set up Vertex AI Model Evaluation with a custom evaluation dataset to measure pass@1 accuracy. The initial evaluation shows 65% pass@1. They want to improve to 80% without collecting more training data. They have already attempted basic prompt engineering (e.g., 'write correct code') with limited improvement. Which approach is most likely to achieve the desired improvement?

Accepted Answer

Use chain-of-thought prompting with few-shot examples of correct code generation.. Chain-of-thought prompting with few-shot examples is the most effective approach because it guides the model through step-by-step reasoning, which is critical for complex code generation tasks. This technique leverages the model's in-context learning ability to improve accuracy without additional training data, directly addressing the need to boost pass@1 from 65% to 80%.

Answer

Reduce the temperature to 0 and set top_p to 1.

Answer

Increase the number of output tokens and enable beam search with width 4.

Answer

Apply reinforcement learning from human feedback (RLHF) using a reward model trained on the existing evaluation dataset.