AI-300 Generative AI Quality Assurance And Observability Practice Test 1 — 15 Questions

Question 1

In an LLM evaluation workflow, what does the 'Coherence' metric measure?

Accepted Answer

The logical flow and structure of the output. Coherence measures how well a model-generated answer makes sense and flows logically as a human-like response.

Answer

The length of the response

Answer

The presence of offensive language

Answer

The factual accuracy of the answer

Question 2

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

Accepted Answer

Content Safety API text analysis. The Content Safety service provides classification categories such as Hate, Self-Harm, Sexual, and Violence to filter model outputs.

Answer

Azure Monitor Logs

Answer

Application Insights Profiler

Answer

Prompt Flow Guardrails

Question 3

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

Accepted Answer

Groundedness. Groundedness specifically assesses whether the generated response is derived from the retrieved documents.

Answer

Relevance

Answer

Performance

Answer

Fluency

Question 4

You are deploying a GenAI app to production and want to track the 'Token Usage' and 'Latency' metrics per user session. Which service should you integrate with your application?

Accepted Answer

Azure Application Insights. Application Insights is the standard observability tool for Azure, enabling custom event tracking for tokens and latency.

Answer

Azure Container Registry

Answer

Azure Key Vault

Answer

Azure Policy

Answer

Azure AI Search

Question 5

When using Prompt Flow for GenAI, where should you store your evaluation results to visualize them over time?

Accepted Answer

Azure Machine Learning workspace. Prompt Flow logs evaluation runs to the Azure Machine Learning workspace, where they can be viewed in the Runs tab.

Answer

GitHub repository

Answer

Azure Blob Storage

Answer

Local file system

Question 6

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

Accepted Answer

Adversarial input classification using a dedicated judge model. Adversarial evaluation using a 'jailbreak' detection model or prompt-based evaluation is the standard approach to identify bypass attempts.

Answer

Token usage tracking

Answer

Latency threshold monitoring

Answer

Semantic similarity scoring

Question 7

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

Accepted Answer

Groundedness evaluator. The Groundedness evaluator in Prompt Flow checks if the response is supported by the context provided.

Answer

Relevance evaluator

Answer

Coherence evaluator

Answer

Fluency evaluator

Question 8

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

Accepted Answer

To serve as a baseline for measuring performance improvements. A Golden Dataset serves as the ground truth to compare model outputs against during automated evaluation runs.

Answer

To increase the model training speed

Answer

To reduce the number of tokens used

Answer

To generate new prompts for users

Question 9

What is the primary function of a 'Prompt Template' in an Azure AI Prompt Flow?

Accepted Answer

To define the structure of the prompt with dynamic inputs. A prompt template allows users to define the structure of the prompt while injecting dynamic variables.

Answer

To cache previous responses

Answer

To perform load balancing

Answer

To automatically retrain the model

Question 10

You are designing a quality assurance gate for your model. If a model output has a 'Violence' score of 0.8 according to Azure AI Content Safety, what is the best practice to handle it?

Accepted Answer

Filter out the output and return a generic refusal. A high score indicates a potential violation, and the output should be blocked or sanitized before reaching the end user.

Answer

Allow the output but log it as an error

Answer

Retrain the model immediately

Answer

Ignore the score if the latency is high

Question 11

You want to measure 'Relevance' in a RAG application. The relevance evaluator detects how well the response answers the user query. If the model provides a factually correct answer that does not address the prompt, which metric will capture this failure?

Accepted Answer

Relevance. Relevance specifically measures if the response directly addresses the user's intent.

Answer

Fluency

Answer

Coherence

Answer

Groundedness

Question 12

Your team wants to perform 'Red Teaming' on your application. Which activity describes this process correctly?

Accepted Answer

Controlled adversarial testing to find safety and security gaps. Red Teaming involves adversarial testing to find edge cases, safety flaws, and security vulnerabilities.

Answer

Unit testing code snippets

Answer

Automated performance testing for latency

Answer

A/B testing two different model versions

Question 13

You are tracking LLM performance. Which metric is most critical to monitor if your cost-per-request is increasing unexpectedly?

Accepted Answer

Token usage. Token usage directly correlates with cost in most LLM provider pricing models.

Answer

User satisfaction score

Answer

Latency

Answer

Model version ID

Question 14

You are using Azure AI Studio to evaluate your model. Which tool allows you to perform batch testing on a large dataset of prompts?

Accepted Answer

Evaluation tab. The 'Evaluation' feature in Azure AI Studio allows for running batch tests using built-in metrics.

Answer

Prompt Playground

Answer

Model catalog

Answer

Deployment logs

Question 15

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Accepted Answer

Custom LLM-as-a-judge evaluation. Custom evaluation using a judge model (LLM-as-a-judge) configured with a rubric allows for checking specific style or brand compliance.

Answer

Token usage threshold alert

Answer

Default coherence metric

Answer

Automated regression testing of code