AI-300 Generative AI Quality Assurance And Observability — 10 Questions

Question 1

In an LLM evaluation workflow, what does the 'Coherence' metric measure?

Accepted Answer

The logical flow and structure of the output. Coherence measures how well a model-generated answer makes sense and flows logically as a human-like response.

Answer

The length of the response

Answer

The presence of offensive language

Answer

The factual accuracy of the answer

Question 2

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

Accepted Answer

Content Safety API text analysis. The Content Safety service provides classification categories such as Hate, Self-Harm, Sexual, and Violence to filter model outputs.

Answer

Azure Monitor Logs

Answer

Application Insights Profiler

Answer

Prompt Flow Guardrails

Question 3

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

Accepted Answer

Groundedness. Groundedness specifically assesses whether the generated response is derived from the retrieved documents.

Answer

Relevance

Answer

Performance

Answer

Fluency

Question 4

You are deploying a GenAI app to production and want to track the 'Token Usage' and 'Latency' metrics per user session. Which service should you integrate with your application?

Accepted Answer

Azure Application Insights. Application Insights is the standard observability tool for Azure, enabling custom event tracking for tokens and latency.

Answer

Azure Container Registry

Answer

Azure Key Vault

Answer

Azure Policy

Answer

Azure AI Search

Question 5

When using Prompt Flow for GenAI, where should you store your evaluation results to visualize them over time?

Accepted Answer

Azure Machine Learning workspace. Prompt Flow logs evaluation runs to the Azure Machine Learning workspace, where they can be viewed in the Runs tab.

Answer

GitHub repository

Answer

Azure Blob Storage

Answer

Local file system

Question 6

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

Accepted Answer

Adversarial input classification using a dedicated judge model. Adversarial evaluation using a 'jailbreak' detection model or prompt-based evaluation is the standard approach to identify bypass attempts.

Answer

Token usage tracking

Answer

Latency threshold monitoring

Answer

Semantic similarity scoring

Question 7

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

Accepted Answer

Groundedness evaluator. The Groundedness evaluator in Prompt Flow checks if the response is supported by the context provided.

Answer

Relevance evaluator

Answer

Coherence evaluator

Answer

Fluency evaluator

Question 8

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

Accepted Answer

To serve as a baseline for measuring performance improvements. A Golden Dataset serves as the ground truth to compare model outputs against during automated evaluation runs.

Answer

To increase the model training speed

Answer

To reduce the number of tokens used

Answer

To generate new prompts for users

Question 9

What is the primary function of a 'Prompt Template' in an Azure AI Prompt Flow?

Accepted Answer

To define the structure of the prompt with dynamic inputs. A prompt template allows users to define the structure of the prompt while injecting dynamic variables.

Answer

To cache previous responses

Answer

To perform load balancing

Answer

To automatically retrain the model

Question 10

You are designing a quality assurance gate for your model. If a model output has a 'Violence' score of 0.8 according to Azure AI Content Safety, what is the best practice to handle it?

Accepted Answer

Filter out the output and return a generic refusal. A high score indicates a potential violation, and the output should be blocked or sanitized before reaching the end user.

Answer

Allow the output but log it as an error

Answer

Retrain the model immediately

Answer

Ignore the score if the latency is high