AI-300 Generative AI Quality Assurance And Observability — 25 Questions

Question 1

When designing a content safety policy, which THREE categories are explicitly supported by the Azure AI Content Safety API?

Accepted Answer

Sexual. Azure AI Content Safety supports Hate, Sexual, Violence, and Self-Harm categories.

Answer

Political bias

Answer

Code syntax errors

Question 2

In an LLM evaluation workflow, what does the 'Coherence' metric measure?

Accepted Answer

The logical flow and structure of the output. Coherence measures how well a model-generated answer makes sense and flows logically as a human-like response.

Answer

The length of the response

Answer

The presence of offensive language

Answer

The factual accuracy of the answer

Question 3

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

Accepted Answer

Content Safety API text analysis. The Content Safety service provides classification categories such as Hate, Self-Harm, Sexual, and Violence to filter model outputs.

Answer

Azure Monitor Logs

Answer

Application Insights Profiler

Answer

Prompt Flow Guardrails

Question 4

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

Accepted Answer

Groundedness. Groundedness specifically assesses whether the generated response is derived from the retrieved documents.

Answer

Relevance

Answer

Performance

Answer

Fluency

Question 5

You are deploying a GenAI app to production and want to track the 'Token Usage' and 'Latency' metrics per user session. Which service should you integrate with your application?

Accepted Answer

Azure Application Insights. Azure Application Insights is the APM service that captures custom metrics, request telemetry, dependencies, and traces, making it the right place to track per-session Token Usage and Latency. You can emit custom metrics (e.g., 'tokens_used', 'latency_ms') tagged with a session ID and visualize them in workbooks or query them with KQL.

Answer

Azure Container Registry

Answer

Azure Key Vault

Answer

Azure Policy

Answer

Azure AI Search

Question 6

When using Prompt Flow for GenAI, where should you store your evaluation results to visualize them over time?

Accepted Answer

Azure Machine Learning workspace. Prompt Flow logs evaluation runs to the Azure Machine Learning workspace, where they can be viewed in the Runs tab.

Answer

GitHub repository

Answer

Azure Blob Storage

Answer

Local file system

Question 7

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

Accepted Answer

Adversarial input classification using a dedicated judge model. Adversarial evaluation using a 'jailbreak' detection model or prompt-based evaluation is the standard approach to identify bypass attempts.

Answer

Token usage tracking

Answer

Latency threshold monitoring

Answer

Semantic similarity scoring

Question 8

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

Accepted Answer

Groundedness evaluator. The Groundedness evaluator in Prompt Flow checks if the response is supported by the context provided.

Answer

Relevance evaluator

Answer

Coherence evaluator

Answer

Fluency evaluator

Question 9

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

Accepted Answer

To serve as a baseline for measuring performance improvements. A Golden Dataset serves as the ground truth to compare model outputs against during automated evaluation runs.

Answer

To increase the model training speed

Answer

To reduce the number of tokens used

Answer

To generate new prompts for users

Question 10

What is the primary function of a 'Prompt Template' in an Azure AI Prompt Flow?

Accepted Answer

To define the structure of the prompt with dynamic inputs. A prompt template allows users to define the structure of the prompt while injecting dynamic variables.

Answer

To cache previous responses

Answer

To perform load balancing

Answer

To automatically retrain the model

Question 11

You are designing a quality assurance gate for your model. If a model output has a 'Violence' score of 0.8 according to Azure AI Content Safety, what is the best practice to handle it?

Accepted Answer

Filter out the output and return a generic refusal. A high score indicates a potential violation, and the output should be blocked or sanitized before reaching the end user.

Answer

Allow the output but log it as an error

Answer

Retrain the model immediately

Answer

Ignore the score if the latency is high

Question 12

You want to measure 'Relevance' in a RAG application. The relevance evaluator detects how well the response answers the user query. If the model provides a factually correct answer that does not address the prompt, which metric will capture this failure?

Accepted Answer

Relevance. Relevance specifically measures if the response directly addresses the user's intent.

Answer

Fluency

Answer

Coherence

Answer

Groundedness

Question 13

Your team wants to perform 'Red Teaming' on your application. Which activity describes this process correctly?

Accepted Answer

Controlled adversarial testing to find safety and security gaps. Red Teaming involves adversarial testing to find edge cases, safety flaws, and security vulnerabilities.

Answer

Unit testing code snippets

Answer

Automated performance testing for latency

Answer

A/B testing two different model versions

Question 14

You are tracking LLM performance. Which metric is most critical to monitor if your cost-per-request is increasing unexpectedly?

Accepted Answer

Token usage. Token usage is the primary driver of LLM cost because most providers bill per input and output token. If cost-per-request rises unexpectedly, monitoring token usage reveals whether prompts, completions, or context windows have grown. This metric directly correlates with the billing unit, making it the most critical to track for cost anomalies.

Answer

User satisfaction score

Answer

Latency

Answer

Model version ID

Question 15

You are using Azure AI Studio to evaluate your model. Which tool allows you to perform batch testing on a large dataset of prompts?

Accepted Answer

Evaluation tab. The Evaluation tab in Azure AI Studio (now Azure AI Foundry) provides batch evaluation capabilities, allowing you to run a model or prompt flow against a large dataset of prompts and compute metrics like groundedness, relevance, and fluency. This is the designated tool for batch testing at scale, as opposed to interactive testing in the playground.

Answer

Prompt Playground

Answer

Model catalog

Answer

Deployment logs

Question 16

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Accepted Answer

Custom LLM-as-a-judge evaluation. Custom evaluation using a judge model (LLM-as-a-judge) configured with a rubric allows for checking specific style or brand compliance.

Answer

Token usage threshold alert

Answer

Default coherence metric

Answer

Automated regression testing of code

Question 17

Which of the following is a key component of an observability strategy for Generative AI applications?

Accepted Answer

Monitoring both technical and quality metrics. Observability requires tracking both technical metrics (latency, errors) and quality metrics (groundedness, safety).

Answer

Hard-coding all model responses

Answer

Using only manual review processes

Answer

Disabling logging to save costs

Question 18

In an LLM pipeline, what is the primary risk of relying solely on automated 'Groundedness' evaluators?

Accepted Answer

They may have false negatives and miss subtle hallucinations. Automated groundedness evaluators (such as the groundedness metric in Azure AI Foundry's evaluation SDK) compare model output against supplied context using a judge model, and that judge is itself probabilistic. Because the evaluator relies on an LLM or NLP model to decide whether a claim is supported, it can return a passing score for a subtly fabricated or misattributed statement. This means false negatives — undetected hallucinations — are an inherent limitation, so human review and complementary metrics (relevance, fluency, retrieval score) remain necessary.

Answer

They prevent the use of custom prompts

Answer

They always increase latency significantly

Answer

They require a permanent database connection

Question 19

You are auditing your model's safety logs and notice several 'jailbreak' attempts. Where can you find these logs in the Azure ecosystem?

Accepted Answer

Log Analytics Workspace. Azure AI Content Safety and Azure OpenAI Service emit diagnostic logs—including jailbreak attempts detected by Prompt Shields—to Azure Monitor. These logs are stored in a Log Analytics workspace, where they can be queried with KQL (e.g., AzureDiagnostics | where Category == 'ContentSafety' or 'Jailbreak'). Log Analytics is the central repository for security and audit logs in Azure, making it the correct place to audit jailbreak attempts.

Answer

Azure Key Vault

Answer

Azure Resource Graph

Answer

Azure Active Directory

Question 20

What is the primary benefit of 'Prompt Versioning' in an MLOps lifecycle?

Accepted Answer

It ensures reproducibility and enables easy rollbacks. Prompt versioning ensures reproducibility by tracking changes to prompts over time, allowing you to revert to a previous version if a new prompt degrades performance. It also enables easy rollbacks, which is critical in MLOps for maintaining stable production systems. It does not directly affect training speed, evaluation necessity, or test generation.

Answer

It increases model training speed

Answer

It eliminates the need for evaluation

Answer

It automatically generates unit tests