Generative AI Quality Assurance And Observability

In an LLM evaluation workflow, what does the 'Coherence' metric measure?

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

You are deploying a GenAI app to production and want to track the 'Token Usage' and 'Latency' metrics per user session. Which service should you integrate with your application?

When using Prompt Flow for GenAI, where should you store your evaluation results to visualize them over time?

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

What is the primary function of a 'Prompt Template' in an Azure AI Prompt Flow?

You are designing a quality assurance gate for your model. If a model output has a 'Violence' score of 0.8 according to Azure AI Content Safety, what is the best practice to handle it?

You want to measure 'Relevance' in a RAG application. The relevance evaluator detects how well the response answers the user query. If the model provides a factually correct answer that does not address the prompt, which metric will capture this failure?

Your team wants to perform 'Red Teaming' on your application. Which activity describes this process correctly?

You are tracking LLM performance. Which metric is most critical to monitor if your cost-per-request is increasing unexpectedly?

You are using Azure AI Studio to evaluate your model. Which tool allows you to perform batch testing on a large dataset of prompts?

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Which of the following is a key component of an observability strategy for Generative AI applications?

In an LLM pipeline, what is the primary risk of relying solely on automated 'Groundedness' evaluators?

You are auditing your model's safety logs and notice several 'jailbreak' attempts. Where can you find these logs in the Azure ecosystem?

What is the primary benefit of 'Prompt Versioning' in an MLOps lifecycle?

You are setting up an evaluation suite for your LLM. Which THREE metrics are commonly provided by the 'Built-in' evaluators in Azure AI Prompt Flow?

Which TWO of the following are recommended methods for identifying 'hallucinations' in a RAG system?

When configuring observability for an AI application, which TWO telemetry types should you collect to analyze both performance and quality?

Which THREE actions are essential when managing a 'Golden Dataset' for LLM evaluation?

When implementing LLM-as-a-judge for evaluation, which TWO factors can influence the reliability of your results?

Which THREE tools in Azure can be used to monitor the health and performance of your GenAI infrastructure?

Which TWO metrics are most useful for evaluating the 'User Experience' in a conversational agent?

You are the MLOps engineer for a customer support chatbot built on Azure OpenAI. Your team wants to continuously monitor the deployed model's responses for harmful content without storing the raw user prompts in your own database. You must implement a solution that evaluates every response in near real time, logs only the safety verdicts and severity scores, and raises an alert when severity exceeds a threshold. Which Azure service should you use to perform this evaluation?

You are evaluating a retrieval-augmented generation (RAG) application that answers questions about internal HR policies. During a quality review, you discover that for some queries the model produces fluent, confident answers that are not supported by the retrieved policy documents. You need to add an automated evaluation that specifically detects whether the response is factually supported by the retrieved context. Which evaluator should you use?

You are preparing a red teaming exercise for a generative AI application built with Azure AI Foundry. The goal is to systematically probe the application for harmful outputs and safety weaknesses before release. Which two activities are appropriate for a red teaming exercise? (Choose two.)

You are building an evaluation pipeline for a generative AI application using the Azure AI evaluation SDK. You want to compare multiple model responses to the same question and obtain a numeric quality score for each response based on a defined criterion. Which evaluator type should you use when you need to score responses against a custom rubric that you define?