Generative AI Quality Assurance And Observability

In an LLM evaluation workflow, what does the 'Coherence' metric measure?

You are monitoring an Azure OpenAI deployment and need to identify if a model is outputting content that violates safety policies. Which Azure AI Content Safety feature should you enable to categorize harmful content?

You are building a RAG application and notice that the model sometimes hallucinates information not present in the retrieved documents. Which evaluation metric should you prioritize to mitigate this?

You are deploying a GenAI app to production and want to track the 'Token Usage' and 'Latency' metrics per user session. Which service should you integrate with your application?

When using Prompt Flow for GenAI, where should you store your evaluation results to visualize them over time?

You need to detect 'jailbreak' attempts in your RAG application. You are implementing a custom evaluation pipeline. Which technique is most effective for identifying adversarial inputs designed to bypass system instructions?

You are evaluating an LLM application using Prompt Flow. You want to measure the 'Groundedness' of the model response relative to the retrieved context. Which evaluator should you configure?

You want to automate the evaluation of your LLM application using a 'Golden Dataset'. What is the primary purpose of this dataset in an MLOps pipeline?

What is the primary function of a 'Prompt Template' in an Azure AI Prompt Flow?

You are designing a quality assurance gate for your model. If a model output has a 'Violence' score of 0.8 according to Azure AI Content Safety, what is the best practice to handle it?

You want to measure 'Relevance' in a RAG application. The relevance evaluator detects how well the response answers the user query. If the model provides a factually correct answer that does not address the prompt, which metric will capture this failure?

Your team wants to perform 'Red Teaming' on your application. Which activity describes this process correctly?

You are tracking LLM performance. Which metric is most critical to monitor if your cost-per-request is increasing unexpectedly?

You are using Azure AI Studio to evaluate your model. Which tool allows you to perform batch testing on a large dataset of prompts?

You want to evaluate how well your model adheres to specific brand guidelines. Which evaluation method is best suited for this?

Which of the following is a key component of an observability strategy for Generative AI applications?

In an LLM pipeline, what is the primary risk of relying solely on automated 'Groundedness' evaluators?

You are auditing your model's safety logs and notice several 'jailbreak' attempts. Where can you find these logs in the Azure ecosystem?

What is the primary benefit of 'Prompt Versioning' in an MLOps lifecycle?

You are setting up an evaluation suite for your LLM. Which THREE metrics are commonly provided by the 'Built-in' evaluators in Azure AI Prompt Flow?

When designing a content safety policy, which THREE categories are explicitly supported by the Azure AI Content Safety API?

Which TWO of the following are recommended methods for identifying 'hallucinations' in a RAG system?

When configuring observability for an AI application, which TWO telemetry types should you collect to analyze both performance and quality?

Which THREE actions are essential when managing a 'Golden Dataset' for LLM evaluation?

When implementing LLM-as-a-judge for evaluation, which TWO factors can influence the reliability of your results?

Which THREE tools in Azure can be used to monitor the health and performance of your GenAI infrastructure?

Which TWO metrics are most useful for evaluating the 'User Experience' in a conversational agent?