Generative AI Quality Assurance And Observability practice questions

Political bias

Why it fails: Not a standard safety category.

Code syntax errors

Why it fails: Not a content safety category.

The length of the response

Why it fails: Coherence scores whether the response flows logically and stays on topic, not its word count; length is measured separately by response-length statistics. Length-based metrics are tempting because they are trivial to compute, and they would be the right measure when evaluating verbosity or token cost rather than logical consistency.

The presence of offensive language

Why it fails: Coherence measures whether the model's output flows logically and stays internally consistent; offensiveness is scored by a separate safety or toxicity metric. It tempts because both assess output quality, but a toxicity metric would be correct when the evaluation goal is detecting harmful or offensive content.

The factual accuracy of the answer

Why it fails: Coherence assesses whether the response flows logically and stays internally consistent, not whether its claims match reality; factual accuracy is measured by groundedness or correctness metrics. It is tempting because both concern answer quality, but a fluent, well-structured response can still contain false statements.

Azure Monitor Logs

Why it fails: Azure Monitor Logs stores and queries telemetry such as requests, tokens and latency; it applies no severity classification to model output text. It is tempting because it centralises monitoring data, but it is the correct choice for diagnostics and alerting on metrics, not for categorising harmful content.

Application Insights Profiler

Why it fails: Application Insights Profiler captures request-level performance traces for latency diagnosis; it performs no classification of text against harm categories. It is tempting because it monitors live application traffic, but it is the correct choice when profiling slow requests, not for categorising violent, hateful or self-harm content.

Prompt Flow Guardrails

Why it fails: Prompt Flow Guardrails apply risk and safety checks within prompt flow orchestration, not to a deployed Azure OpenAI endpoint's live output. It is tempting because it addresses safety, but it is the correct choice when building flows that need input/output validation, not for categorising violations from an existing deployment.

Relevance

Why it fails: Relevance scores how well retrieved context matches the query, not whether the answer stays grounded in that context, so fabricated content can still rank as relevant. Relevance is the correct metric when tuning retrieval quality or ranking passages.

Performance

Why it fails: Performance measures latency and throughput, not whether generated text is grounded in retrieved documents. It is tempting because performance tuning matters when a RAG pipeline is slow or costly at scale, making it the right focus for optimisation work. Here, however, hallucination requires a groundedness or faithfulness metric that scores answer claims against the retrieved context.

Fluency

Why it fails: Fluency measures grammatical and linguistic naturalness, so a fluent hallucination scores well and the metric cannot detect unsupported claims. Fluency evaluation is the right choice when assessing generated text readability or comparing language quality of model outputs.

Azure Container Registry

Why it fails: Azure Container Registry stores and distributes container images; it records no per-session token or latency data. It is tempting because GenAI apps are often containerised, and would be correct when hosting and versioning the image your deployment pulls.

Azure Key Vault

Why it fails: Azure Key Vault stores and rotates secrets, keys and certificates; it emits no token-usage or latency telemetry per session. It is tempting because GenAI apps retrieve API keys from it, and would be correct when securing model credentials rather than monitoring runtime metrics.

Azure Policy

Why it fails: Azure Policy evaluates and enforces resource configuration compliance; it captures no runtime token or latency telemetry. It is tempting because governance is part of production readiness, and would be correct when restricting which SKUs or regions deployments may use.

Azure AI Search

Why it fails: Azure AI Search indexes and queries documents; it does not collect per-session token or latency telemetry. It is tempting because it integrates with GenAI retrieval pipelines, and would be correct when grounding model responses in your own indexed content.

GitHub repository

Why it fails: A GitHub repository stores code and version history; it offers no native Prompt Flow run metrics or time-series visualisation. It is tempting as a familiar place to keep evaluation artefacts, and would suit sharing prompt source files, but visualising metrics over time requires a workspace-linked store.

Azure Blob Storage

Why it fails: Azure Blob Storage holds raw artefacts but provides no native metric tracking or run comparison views, so Prompt Flow cannot chart evaluation trends from it. It is tempting as cheap durable storage, and would suit archiving result files, but visualising metrics over time requires a workspace-linked store.

Local file system

Why it fails: The local file system keeps results on one machine, so Prompt Flow cannot aggregate runs or render metric trends across evaluations. It is tempting for quick offline experiments, and would suffice for a single developer's scratch runs, but shared, longitudinal visualisation needs a cloud-backed store.

Token usage tracking

Why it fails: Token usage tracking measures prompt and completion length, which correlates with cost and verbosity, not adversarial intent; a jailbreak can be short or long. It is tempting because unusual token counts sometimes flag anomalous prompts, but detecting instruction-bypass attempts requires analysing semantic intent, such as classifier or prompt-injection scoring.

Latency threshold monitoring

Why it fails: Latency threshold monitoring detects slow responses, which reflects model or infrastructure performance rather than whether an input attempts to bypass system instructions. It is tempting because adversarial prompts can occasionally trigger longer processing, but jailbreak detection needs content-level analysis of the input, not timing metrics.

Semantic similarity scoring

Why it fails: Semantic similarity scoring compares meaning between texts, so it measures topical closeness rather than whether an input is crafted to override system instructions. It is tempting because it is used for RAG relevance and groundedness checks, but jailbreak detection requires identifying adversarial intent, typically via a dedicated classifier or prompt-injection detector.

Relevance evaluator

Why it fails: Relevance scores how pertinent the response is to the question, not whether its claims are supported by the retrieved context. Tempting because relevance and groundedness both compare response against context, and relevance would be correct when judging whether retrieved passages address the user's query.

Coherence evaluator

Why it fails: Coherence judges logical flow and structure between sentences, not whether statements are entailed by the retrieved context. Tempting because it is a common quality metric in Prompt Flow, and it would be correct when evaluating how well the response reads as a connected argument rather than its factual support.

Fluency evaluator

Why it fails: Fluency measures grammatical and linguistic quality of the generated text, with no reference to the retrieved context, so it cannot detect unsupported claims. Tempting because it is a standard Prompt Flow evaluator, and it would be correct when assessing readability of model output rather than factual grounding.

To increase the model training speed

Why it fails: A Golden Dataset exists to score outputs against known-correct references, not to accelerate training; it is typically never fed to the model at all. Training speed depends on batch size, hardware and optimiser settings. The dataset would be the right tool when you need regression testing or benchmark scoring of prompt or model changes.

To reduce the number of tokens used

Why it fails: Datasets do not affect token consumption.

To generate new prompts for users

Why it fails: A Golden Dataset is fixed reference data used to score responses, not a generator of user-facing prompts. Prompt generation belongs to prompt templates or the application layer. It would be the right artefact when you need to validate that a prompt change has not degraded answer quality against known expected results.

To cache previous responses

Why it fails: Caching stores prior responses to avoid recomputation and is provided by caching layers or the service, not by a prompt template. A prompt template defines the parameterised text and system message submitted to the model, making it the right component when prompts must be reused consistently across flow runs.

To perform load balancing

Why it fails: Load balancing distributes requests across compute instances and is handled by infrastructure or an endpoint, not by a prompt template. A prompt template instead defines the structured text, system instructions and variable placeholders sent to the model, so it would be the right construct when the task is to standardise and parameterise model input.

To automatically retrain the model

Why it fails: Retraining updates model weights from training data and is performed through training jobs or pipelines, not through prompt authoring. A prompt template structures the input text and injects variables at runtime, so it is the correct tool when the goal is consistent, reusable prompt construction rather than changing the underlying model.

Allow the output but log it as an error

Why it fails: Logging while allowing the output still delivers violent content to the user, so the gate fails its purpose. This is tempting because logging preserves auditability and suits low-severity scores, but 0.8 sits in the high band where Azure AI Content Safety expects blocking or rejection, not passive recording.

Retrain the model immediately

Why it fails: A 0.8 Violence score is high severity, so the gate should block or flag the output rather than retrain. Retraining is tempting because it addresses underlying model behaviour, and it would be right for systematic bias found across many outputs, but a single scored response needs immediate filtering, not a costly training cycle.

Ignore the score if the latency is high

Why it fails: Latency is unrelated to content severity; a 0.8 Violence score must trigger blocking regardless of response time. Ignoring it is tempting when performance budgets are tight, and latency thresholds are legitimate for routing or caching decisions, but they never override safety gating on harmful content.

Fluency

Why it fails: Fluency scores grammatical and linguistic quality of the generated text, so a well-formed answer that ignores the prompt still scores highly. It is tempting because poor wording often accompanies poor answers, but fluency would be the correct metric when evaluating readability rather than query alignment.

Coherence

Why it fails: Coherence measures whether the response is logically structured and internally consistent, not whether it addresses the user's question. A factually correct but off-topic answer can read perfectly coherently. Coherence would be the right metric when assessing logical flow across sentences in a multi-turn answer.

Groundedness

Why it fails: Groundedness checks whether claims are supported by the retrieved context, so a factually correct answer drawn from the sources passes even when it misses the prompt. It is tempting because both concern answer quality, but groundedness is correct when detecting hallucination against retrieved documents.

Unit testing code snippets

Why it fails: Red teaming probes a deployed AI system adversarially to expose harmful, unsafe or jailbreak outputs; unit testing verifies individual code functions against expected results during development. Unit testing is the right choice for validating deterministic logic, but it exercises no model behaviour and cannot surface prompt-injection or content-safety failures.

Automated performance testing for latency

Why it fails: Red teaming is adversarial simulation that probes defences by mimicking attacker techniques, not measuring latency. Performance testing is tempting because it also exercises the application under load, but it belongs to non-functional or load-testing scenarios, where throughput and response times, not exploitation of weaknesses, are the objective.

A/B testing two different model versions

Why it fails: Red teaming adversarially attacks an application to uncover exploitable weaknesses; A/B testing compares model versions against business or quality metrics. A/B testing is tempting because it evaluates deployed models, but it belongs to experimentation and MLOps scenarios, where statistical comparison of variants, not adversarial exploitation, is the goal.

User satisfaction score

Why it fails: User satisfaction is a subjective quality measure and does not reveal the token counts, model tier, or retry behaviour driving cost-per-request upward. It is tempting because satisfaction is a headline LLM metric, yet it responds to answer quality rather than to the billing dimensions that determine spend.

Latency

Why it fails: Latency measures response time, not token consumption or request volume, so rising latency does not explain a cost-per-request increase. It is tempting because latency is a core performance signal and often correlates with larger prompts or outputs, but it is a symptom rather than the cost driver itself.

Model version ID

Why it fails: Model version ID is an identifier, not a measurement, so it cannot show why cost-per-request is rising; it only labels which deployment served traffic. It is tempting because a version change can coincide with a price or token-usage shift, but the identifier itself carries no cost signal to monitor.

Prompt Playground

Why it fails: Prompt Playground submits one prompt at a time for interactive comparison, so it cannot iterate a large prompt dataset and aggregate metrics. It is tempting because it is the natural place to draft and refine prompts manually, which suits ad-hoc prompt engineering rather than scheduled batch evaluation.

Model catalog

Why it fails: The model catalog is a browsable library of foundation models for selection and deployment, not an evaluation harness. It is the right place to find a base model, whereas batch evaluation of prompts requires the evaluation or flow tooling.

Deployment logs

Why it fails: Deployment logs record inference requests, latency and errors from a deployed endpoint; they do not execute prompts against a dataset or score outputs. They are tempting because they surface real traffic for monitoring, which suits production observability rather than the batch evaluation runs Azure AI Studio's evaluation flow provides.

Token usage threshold alert

Why it fails: Token usage thresholds monitor cost and quota consumption, not output quality or brand alignment. It tempts because token metrics are readily available in the portal and easy to alert on, yet they describe volume rather than content, so they cannot detect violations of tone, terminology or style rules.

Default coherence metric

Why it fails: Coherence measures whether generated text flows logically and reads naturally; it says nothing about brand terminology, tone or prohibited phrasing. It tempts because it is a built-in, zero-configuration metric, but brand adherence demands a custom evaluator or human review scored against the organisation's specific guidelines.

Automated regression testing of code

Why it fails: Regression testing verifies that code changes have not broken existing software behaviour; it inspects program logic, not model outputs against brand tone, terminology or style rules. It tempts because evaluation pipelines are automated like test suites, yet brand adherence requires a custom evaluator or human rubric scoring the generated content itself.

Hard-coding all model responses

Why it fails: Hard-coding responses removes the telemetry, prompt/response logging and drift signals that observability depends on, so failures cannot be traced or evaluated. It is tempting because fixed responses guarantee deterministic output for regression testing, which suits static unit tests rather than production generative AI monitoring.

Using only manual review processes

Why it fails: Manual review cannot scale to production traffic volumes or provide continuous metrics, traces and alerting across model, retrieval and prompt layers. Tempting because human evaluation catches quality issues automated checks miss, but it supplements rather than constitutes an observability strategy.

Disabling logging to save costs

Why it fails: Observability depends on telemetry, traces and prompt/response logs; disabling logging removes the evidence needed to detect drift, latency and failures. Tempting as a cost-control measure, but cost reduction is achieved through sampling and retention policies, not by eliminating the data pipeline entirely.

They prevent the use of custom prompts

Why it fails: Groundedness evaluators score responses against retrieved context and do not restrict prompt construction, so custom prompts remain usable. The genuine risk is that automated scoring can be gamed or misjudge context; these evaluators suit high-volume regression testing, not sole production gating.

They always increase latency significantly

Why it fails: Latency is a performance consideration, not the primary risk; groundedness evaluators can run asynchronously or be sampled. The real risk is that automated scoring can be gamed or misjudge context, so they suit high-volume regression testing rather than sole gating of production responses.

They require a permanent database connection

Why it fails: Groundedness evaluators compare responses against supplied context and need no permanent database connection, so this is not the risk. The actual risk is that automated scoring can be gamed or misjudge context; such evaluators suit high-volume regression testing rather than sole production gating.

Azure Key Vault

Why it fails: Key Vault stores and controls access to secrets, keys and certificates; it does not capture model prompt or safety telemetry. It would be correct for retrieving an API key or connection secret used by the application, but jailbreak attempts appear in Azure Monitor diagnostic logs from the AI resource.

Azure Resource Graph

Why it fails: Resource Graph queries Azure Resource Manager metadata — resource types, properties, tags and configuration — not runtime prompt content. It would suit inventory questions such as listing AI resources across subscriptions, whereas jailbreak attempts require Azure Monitor diagnostic logs emitted by the model deployment.

Microsoft Entra ID

Why it fails: Jailbreak attempts are logged by Azure AI Content Safety and surfaced through Azure Monitor diagnostic logs, not by an identity provider. Microsoft Entra ID records authentication and sign-in events, so it would be the right place to audit suspicious logins, not prompt-injection or jailbreak detections.

It increases model training speed

Why it fails: Prompt versioning records prompt changes for traceability and reproducibility; it does not alter model training, which depends on data, hyperparameters and compute. It is tempting because faster iteration on prompts can shorten the overall development cycle, so the speed gain is misattributed to the versioning feature itself.

It eliminates the need for evaluation

Why it fails: Prompt versioning supports comparison of prompt variants but cannot remove evaluation; each version still requires scoring against datasets to detect regressions. It is tempting because versioning makes A/B comparison between prompts straightforward, leading teams to assume evaluation becomes unnecessary once versions are tracked.

It automatically generates unit tests

Why it fails: Prompt versioning tracks and compares prompt iterations, enabling rollback and regression analysis; it does not generate unit tests, which come from test frameworks in the CI pipeline. It is tempting because versioned prompts feed evaluation suites, so teams assume the versioning mechanism itself produces the test artefacts.