Hard Difficulty Questions

Implement custom safety filters on the model outputs

Why wrong: Safety filters address content but not transparency and auditability.

Use a larger context window to capture all interactions

Why wrong: Context window size is unrelated to transparency requirements.

Train an internal model from scratch to replace the vendor model

Why wrong: Replacing the model is a major undertaking and not a critical step for compliance with an existing vendor model.

Private Google Access for on-premises connectivity

Why wrong: Private Google Access addresses network privacy, not chatbot deployment or compliance.

Dialogflow CX with Cloud Logging

Why wrong: Dialogflow CX is powerful but may not provide the same level of gen AI customization and auditability.

Cloud AI Platform Pipelines

Why wrong: Cloud AI Platform Pipelines is for ML pipeline orchestration, not chatbot deployment.

Use only open-source models to ensure transparency

Why wrong: Open-source is not required by law.

Automatically delete all user logs after 30 days

Why wrong: GDPR does not specify a fixed retention period; it depends on purpose.

Confidence indicators and model cards

Why wrong: Confidence indicators show certainty but don't provide source citations; model cards are high-level documentation, not per-output explanations.

Content safety filters and human oversight

Why wrong: Safety filters block harmful content but don't provide source citations or reasoning transparency.

Datasheets for Datasets and PAIR Explorables

Why wrong: These are for dataset documentation and interactive exploration, not for per-output explainability.

Increase the batch size for batch requests

Why wrong: Batching reduces per-request overhead but does not reduce the number of tokens processed per request.

Switch to a smaller model size

Why wrong: Smaller models may reduce per-token cost but can produce lower quality; the prompt length issue remains.

Ignore the cost increase as it will stabilize

Why wrong: Ignoring cost is not a strategy; the issue should be addressed proactively.

Reduce the temperature to 0 and set top_p to 1.

Why wrong: Deterministic sampling reduces randomness but does not enhance reasoning.

Increase the number of output tokens and enable beam search with width 4.

Why wrong: Beam search can improve likelihood but not necessarily reasoning quality.

Apply reinforcement learning from human feedback (RLHF) using a reward model trained on the existing evaluation dataset.

Why wrong: RLHF requires new human preference data, not just the evaluation dataset.

Use a zero-shot model with context in prompt

Why wrong: Zero-shot with context is limited by context length and may not scale.

Fine-tune the model on the documents

Why wrong: Fine-tuning embeds data into the model, risking exposure.

Use prompt engineering to instruct the model

Why wrong: Prompt engineering alone cannot handle large volumes of proprietary data effectively.

Vertex AI Grounding with their compliance database.

Why wrong: Grounding improves accuracy but does not block specific phrases.

Prompt engineering with instructions to avoid guarantees.

Why wrong: Prompt engineering is not foolproof; safety filters are more robust.

Reinforcement learning from human feedback (RLHF) on the model.

Why wrong: RLHF is costly and not a real-time filter.

Build a retrieval-augmented generation (RAG) system in Vertex AI Agent Builder

Why wrong: RAG is for answering questions over documents, not for structured extraction of specific clauses. Agent Builder is more suited for conversational interfaces.

Use the Gemini API directly with the raw PDF files as input

Why wrong: Gemini API can accept PDFs, but without preprocessing, the model may not correctly interpret complex layouts, leading to errors. Cost may also be higher for large PDFs.

Fine-tune a Gemini model on 100 annotated contracts and run inference on all contracts

Why wrong: 100 contracts are too few to handle layout variations; fine-tuning is expensive and may not generalize well to diverse document formats.

Persistent Disk

Why wrong: Persistent Disk is block storage but not as performant for massive training data access patterns.

Cloud Filestore

Why wrong: Filestore is a file storage service but may not match Cloud Storage's throughput for training data.

Cloud Spanner

Why wrong: Cloud Spanner is a relational database, not designed for training data storage.

Using only English prompts

Why wrong: Limiting to one language does not reduce bias; it may introduce language-related bias.

Increasing model size

Why wrong: Larger models may learn more biases from training data; size does not inherently reduce bias.

Fine-tune the model with RLHF

Why wrong: RLHF helps alignment but doesn't guarantee factual grounding.

Set the temperature to 0.0

Why wrong: Low temperature reduces randomness but doesn't prevent hallucination.

Use prompt engineering to instruct the model to be accurate

Why wrong: Prompt engineering is insufficient for factual accuracy.

Use a zero-shot prompt asking for a patient note in bullet points

Why wrong: Zero-shot without examples may not consistently follow the template; bullet points may not match the required sections.

Set temperature to 0 and max tokens to a high value

Why wrong: Low temperature reduces randomness but does not enforce a specific template structure without examples.

Use a system prompt that lists the sections as instructions

Why wrong: System prompts with instructions alone may be less reliable than few-shot examples for complex formatting.

Migrate all GenAI workloads to a single on-premises server to reduce cloud costs

Why wrong: On-premises may reduce cloud costs but limits access to managed services and scalability, hindering innovation.

Mandate all teams use a single model (e.g., Gemini) via a centralized Vertex AI endpoint with usage quotas

Why wrong: A single model may not fit all use cases, limiting innovation and effectiveness.

Allow teams to continue using their own models but require them to submit monthly cost reports

Why wrong: Monthly reports are reactive and do not proactively control costs or ensure consistency.

Switch to a smaller model like Gemini 1.5 Flash to reduce latency.

Why wrong: While this might improve latency, it does not ensure the knowledge base is properly used; incorrect answers may persist.

Increase the context window of the model to allow more knowledge base content.

Why wrong: Longer context can exacerbate latency and does not fix retrieval or grounding issues.

Fine-tune the Gemini model with the company's historical chat logs to improve domain-specific responses.

Why wrong: Fine-tuning is expensive and may not resolve the grounding issue; the model might still ignore the knowledge base.

Add more diverse training data

Why wrong: May not address existing biases without analysis.

Manually review all generated emails before sending

Why wrong: Not scalable for large volumes.

Switch to a different generative model

Why wrong: Other models may also have bias.

The safety filters are too aggressive; relax them

Why wrong: Safety filters are not causing this bias; relaxing them would not address the stereotypical language.

The temperature parameter is set too low; increase it to introduce more randomness

Why wrong: Temperature affects creativity, not systematic bias in language sophistication.

The model architecture is biased; switch to a different base model

Why wrong: The architecture is not the primary cause; the bias is in the training data, not the model type.

Increase the model's temperature to make outputs more conservative.

Why wrong: Higher temperature increases randomness, making outputs less predictable and potentially more toxic.

Use a separate language model to rephrase the outputs before sending to users.

Why wrong: This adds latency and complexity; not the most immediate or direct measure.

Fine-tune the model on a curated dataset of polite conversations.

Why wrong: Fine-tuning is effective but takes time and resources; not an immediate fix.

Safety filters: Pro has stricter safety defaults

Why wrong: Safety settings are configurable for both.

Multimodal capability: Flash does not support image input

Why wrong: Both support multimodal input.

Deploy the model with a larger max_output_tokens

Why wrong: Max tokens does not affect hallucination or consistency.

Use prompt engineering with few-shot examples

Why wrong: Few-shot may help but not as effective as fine-tuning for consistency.

Increase the temperature to 0.9

Why wrong: Higher temperature increases randomness, making outputs less consistent.