AI-300 Generative AI Quality Assurance And Observability Practice Test 2 — 15 Questions

Question 1

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Accepted Answer

Custom LLM-as-a-judge evaluation. Custom evaluation using a judge model (LLM-as-a-judge) configured with a rubric allows for checking specific style or brand compliance.

Answer

Token usage threshold alert

Answer

Default coherence metric

Answer

Automated regression testing of code

Question 2

Which of the following is a key component of an observability strategy for Generative AI applications?

Accepted Answer

Monitoring both technical and quality metrics. Observability requires tracking both technical metrics (latency, errors) and quality metrics (groundedness, safety).

Answer

Hard-coding all model responses

Answer

Using only manual review processes

Answer

Disabling logging to save costs

Question 3

In an LLM pipeline, what is the primary risk of relying solely on automated 'Groundedness' evaluators?

Accepted Answer

They may have false negatives and miss subtle hallucinations. Automated groundedness evaluators (such as the groundedness metric in Azure AI Foundry's evaluation SDK) compare model output against supplied context using a judge model, and that judge is itself probabilistic. Because the evaluator relies on an LLM or NLP model to decide whether a claim is supported, it can return a passing score for a subtly fabricated or misattributed statement. This means false negatives — undetected hallucinations — are an inherent limitation, so human review and complementary metrics (relevance, fluency, retrieval score) remain necessary.

Answer

They prevent the use of custom prompts

Answer

They always increase latency significantly

Answer

They require a permanent database connection

Question 4

You are auditing your model's safety logs and notice several 'jailbreak' attempts. Where can you find these logs in the Azure ecosystem?

Accepted Answer

Log Analytics Workspace. Azure AI Content Safety and Azure OpenAI Service emit diagnostic logs—including jailbreak attempts detected by Prompt Shields—to Azure Monitor. These logs are stored in a Log Analytics workspace, where they can be queried with KQL (e.g., AzureDiagnostics | where Category == 'ContentSafety' or 'Jailbreak'). Log Analytics is the central repository for security and audit logs in Azure, making it the correct place to audit jailbreak attempts.

Answer

Azure Key Vault

Answer

Azure Resource Graph

Answer

Azure Active Directory

Question 5

What is the primary benefit of 'Prompt Versioning' in an MLOps lifecycle?

Accepted Answer

It ensures reproducibility and enables easy rollbacks. Prompt versioning ensures reproducibility by tracking changes to prompts over time, allowing you to revert to a previous version if a new prompt degrades performance. It also enables easy rollbacks, which is critical in MLOps for maintaining stable production systems. It does not directly affect training speed, evaluation necessity, or test generation.

Answer

It increases model training speed

Answer

It eliminates the need for evaluation

Answer

It automatically generates unit tests

Question 6

You are setting up an evaluation suite for your LLM. Which THREE metrics are commonly provided by the 'Built-in' evaluators in Azure AI Prompt Flow?

Accepted Answer

Coherence. Azure AI Prompt Flow's built-in evaluators include Coherence (A), which measures how logically and naturally the generated text flows; Groundedness (B), which checks whether the response is supported by the provided source/context; and Relevance (C), which assesses how well the answer addresses the user's question. These three are standard quality metrics exposed by the built-in evaluation flow for assessing LLM outputs. Deployment cost (D) is an operational/cost metric, not a built-in evaluator metric, and training duration (E) relates to model training rather than response evaluation, so neither belongs here.

Answer

Deployment cost

Answer

Training duration

Question 7

Which TWO of the following are recommended methods for identifying 'hallucinations' in a RAG system?

Accepted Answer

Using an LLM-as-a-judge to compare against ground truth. Option A is correct because using an LLM-as-a-judge to compare the generated answer against a known ground-truth reference is a standard evaluation technique for detecting hallucinations, as it directly assesses factual consistency and correctness of the output. Option C is correct because comparing the generated output against the retrieval context verifies whether the model's claims are actually grounded in the retrieved documents, which is the core definition of hallucination detection in RAG (faithfulness/groundedness checking). Option B is not appropriate because grammatical errors indicate fluency or language quality issues, not factual fabrication, and a hallucination can be perfectly grammatical. Option D is not appropriate because response speed is a latency/performance metric and has no bearing on factual accuracy. Option E is not appropriate because counting prompt tokens measures input size and cost, not whether the generated content is factually supported.

Answer

Checking for grammatical errors

Answer

Measuring model response speed

Answer

Counting total prompt tokens

Question 8

When configuring observability for an AI application, which TWO telemetry types should you collect to analyze both performance and quality?

Accepted Answer

Content safety flags. Option B (Content safety flags) is correct because content safety flags are a quality-oriented telemetry signal that captures whether the AI application's generated or processed content violates safety policies, enabling analysis of output quality and responsible-AI compliance. Option E (Request latency) is correct because request latency is a core performance telemetry metric that measures the time taken to serve each request, allowing you to analyze responsiveness, throughput, and bottlenecks in the AI application. The remaining options do not belong: hard drive space (A) and CPU temperature (D) are infrastructure/host-level metrics that do not directly describe AI application performance or output quality, and user profile pictures (C) are user data with no telemetry value for performance or quality analysis.

Answer

Hard drive space

Answer

User profile pictures

Answer

CPU temperature

Question 9

Which THREE actions are essential when managing a 'Golden Dataset' for LLM evaluation?

Accepted Answer

Versioning the dataset to track changes. Option A (Versioning the dataset to track changes) is correct because a golden dataset must be immutable and auditable, so semantic versioning or hashes (e.g., Git, DVC, or dataset version tags) let you reproduce evaluation results and correlate score changes with specific dataset revisions. Option D (Updating the dataset as model capabilities change) is correct because as models improve, static benchmarks saturate; curating harder or newly relevant cases (e.g., new task types, edge cases, or regressions) keeps the evaluation discriminative and aligned with current deployment goals. Option E (Curating high-quality prompt-response pairs) is correct because the golden dataset's reference answers must be accurate, unambiguous, and representative, typically validated by domain experts, since noisy or biased ground truth invalidates every metric computed against it. Option B is not essential because encryption with public keys only is an asymmetric-encryption detail, not a dataset-management requirement; confidentiality is handled by access controls and, if needed, symmetric encryption or KMS. Option C is not essential because deleting logs after each test run destroys the audit trail and reproducibility evidence needed to compare runs and debug regressions.

Answer

Encrypting the data with public keys only

Answer

Deleting logs after each test run

Question 10

When implementing LLM-as-a-judge for evaluation, which TWO factors can influence the reliability of your results?

Accepted Answer

The specific prompt instructions provided to the judge LLM. Option B is correct because the prompt instructions given to the judge LLM directly shape its evaluation criteria, rubric interpretation, and output format, so vague or biased instructions produce inconsistent, unreliable judgments. Option E is correct because the judge LLM's own capability—its reasoning ability, context window, training quality, and susceptibility to position or verbosity bias—bounds how accurately it can assess responses, making a weaker judge less reliable than a stronger one. Options A, C, and D do not belong: dashboard color is purely cosmetic, workspace token quota affects capacity/cost rather than judgment reliability, and storage account location affects latency, residency, and compliance but not the judge's evaluation quality.

Answer

The color of the application dashboard

Answer

The number of tokens available in the workspace quota

Answer

The location of the storage account

Question 11

Which THREE tools in Azure can be used to monitor the health and performance of your GenAI infrastructure?

Accepted Answer

Azure Log Analytics. Azure Log Analytics (B) is correct because it is the workspace-based service that ingests and queries telemetry such as logs and metrics from GenAI resources using Kusto Query Language (KQL), enabling health and performance analysis. Azure Application Insights (C) is correct because it provides application performance monitoring (APM) for GenAI apps, capturing request rates, latency, failures, dependency calls to model endpoints, and distributed traces. Azure Monitor (D) is correct because it is the umbrella monitoring platform that collects metrics, activity logs, and diagnostics, and drives alerts, dashboards, and autoscale for GenAI infrastructure. Azure Front Door (A) is a global HTTP load balancer and CDN for web traffic, not a health/performance monitoring tool, and Microsoft Entra ID (E) is an identity and access management service, so neither belongs here.

Answer

Azure Front Door

Answer

Azure Active Directory

Question 12

Which TWO metrics are most useful for evaluating the 'User Experience' in a conversational agent?

Accepted Answer

Relevance. Relevance (A) is a core user-experience metric for a conversational agent because it measures whether the agent's responses actually address the user's intent, which directly determines perceived usefulness and satisfaction. Latency (C) is equally essential since response time governs conversational flow; high latency makes interactions feel sluggish and breaks the natural turn-taking users expect. Disk I/O speed (B) is an infrastructure metric that affects backend performance but is not a direct measure of how users experience the conversation. Model training loss (D) is an offline optimization signal describing how well the model fits training data, not the quality of live interactions. GPU voltage (E) is a hardware-level electrical measurement with no bearing on conversational user experience.

Answer

Disk I/O speed

Answer

Model training loss

Answer

GPU voltage

Question 13

You are the MLOps engineer for a customer support chatbot built on Azure OpenAI. Your team wants to continuously monitor the deployed model's responses for harmful content without storing the raw user prompts in your own database. You must implement a solution that evaluates every response in near real time, logs only the safety verdicts and severity scores, and raises an alert when severity exceeds a threshold. Which Azure service should you use to perform this evaluation?

Accepted Answer

Azure AI Content Safety. Azure AI Content Safety is purpose-built to analyze generative AI inputs and outputs for harmful categories, returning per-category severity scores in near real time. Because it evaluates content without requiring you to persist raw prompts, it satisfies both the privacy constraint and the monitoring requirement. Its severity scores can be routed to Azure Monitor to trigger alerts when thresholds are exceeded.

Answer

Azure Monitor Application Insights custom events

Answer

Azure Machine Learning model data collector

Answer

Azure AI Language Personally Identifiable Information detection

Question 14

You are evaluating a retrieval-augmented generation (RAG) application that answers questions about internal HR policies. During a quality review, you discover that for some queries the model produces fluent, confident answers that are not supported by the retrieved policy documents. You need to add an automated evaluation that specifically detects whether the response is factually supported by the retrieved context. Which evaluator should you use?

Accepted Answer

Groundedness evaluator. Groundedness evaluation directly compares a model's response against the retrieved context to determine whether claims are supported. Because the observed problem is confident but unsupported answers, this evaluator targets the exact failure mode. Fluency, relevance, and coherence measure other quality dimensions and would not reliably flag responses that are well-written but not grounded in the source documents.

Answer

Relevance evaluator

Answer

Fluency evaluator

Answer

Coherence evaluator

Question 15

You are preparing a red teaming exercise for a generative AI application built with Azure AI Foundry. The goal is to systematically probe the application for harmful outputs and safety weaknesses before release. Which two activities are appropriate for a red teaming exercise? (Choose two.)

Accepted Answer

Crafting adversarial prompts that attempt to elicit prohibited content across categories such as violence, hate, and self-harm.. Red teaming is an adversarial process that probes a generative AI application for safety weaknesses. Crafting adversarial prompts across harm categories and running automated adversarial prompt datasets through Azure AI Foundry risk and safety evaluations both directly test whether the application can be induced to produce harmful outputs. The other activities concern performance, access control, or text similarity and do not evaluate safety behavior.

Answer

Calculating the BLEU score between model responses and a reference translation dataset.

Answer

Measuring the average token latency of the model across a representative production workload.

Answer

Reviewing the application's role-based access control assignments for the deployment resource.