CCNA Evaluation and Monitoring Questions

Compliance-grade traceability demands durable, queryable records of each request, the context supplied to the model, the response, and the serving model version. Unity Catalog inference tables store exactly that as governed Delta tables, so SQL queries and joins to model version metadata answer audit questions months after the traffic occurred. Client logs, offline experiment runs, and aggregate dashboards all lack either the payload detail or the retention and governance needed.

The drop begins exactly when the index was rebuilt, and relevance remains flat, which isolates the change to the evidence supplied to the generator rather than to the prompt, the LLM, or the judge. Degraded retrieval after re-indexing, from altered chunking, embedding mismatches, or an incomplete build, is the best-supported conclusion and the right place to investigate first.

Groundedness directly assesses whether the response is supported by the retrieved context, so a pattern of ignoring documents and using stale base-model knowledge will show up as low groundedness. That makes it the metric that quantifies the behavior the team is investigating in their catalog assistant.

The rising scores reflect a judge that rewards fluency because it shares a model family with the assistant and was given a style-oriented rubric, so the instrument itself is blind to factual omission. Restoring detection requires redefining the rubric around factual coverage of the source, anchoring it with human-labeled examples, and monitoring judge-human agreement so drift in the judge is caught before it masks another regression.

Fluctuating judge scores usually stem from nondeterministic sampling and drifting judge model versions. Pinning the judge version and setting temperature to zero makes each scoring pass reproducible, and validating the judge against human labels confirms that the stabilized scores are also accurate enough to gate releases reliably.

To capture retrieved documents, the final prompt, and the model response in a single trace, the engineer should use MLflow Tracing with manual instrumentation. Spans for each step share a trace context, providing a unified view. This is the native Databricks approach for debugging RAG applications and integrates with Mosaic AI Model Serving for production monitoring.

Lakehouse Monitoring is designed to monitor Delta tables, including inference tables, for data quality and drift. By creating a monitor with a custom metric for response length, the team can automatically detect significant changes from the baseline and receive alerts. This is the most direct and integrated approach on Databricks for this monitoring requirement.

Faithfulness measures whether the generated answer is derived exclusively from the retrieved context. If a model generates information not supported by the source text, it is 'hallucinating' or ignoring the context. Monitoring faithfulness is crucial for RAG systems because it directly detects when the model drifts away from the ground truth provided by the internal knowledge base, which is the primary value proposition of a RAG architecture.

A defensible promotion gate requires a controlled comparison where the prompt template is the only variable, which means running both templates over the same dataset with the judge model and retrieval configuration pinned, and inspecting per-question deltas instead of relying on averages that can mask localized regressions. Together these practices attribute any score movement to the change under test and make the decision reproducible.

Groundedness evaluation compares generated answers against retrieved context, which is precisely the failure mode described. Running it as a scheduled job over recent inference-table records produces a quantified, recurring metric, and persisting results to Delta supports alerting on threshold breaches. Operational metrics and manual review cannot detect semantically unsupported but fluent responses.

Reproducibility depends on capturing inputs and configuration, not just scores. Logging the run with parameters, the prompt template, and a versioned dataset as artifacts and metadata ties every result to the exact conditions that produced it. That lets anyone rerun or audit the evaluation later, even after library upgrades or dataset changes.

Inference tables are the Databricks mechanism that logs request and response payloads for Mosaic AI Model Serving endpoints. Because the logged payloads include the user question, retrieved context, and generated answer, the team gains the raw material needed to run MLflow LLM Evaluation or custom monitoring jobs later against real production traffic.

Groundedness measures whether the answer is supported by the retrieved context, so a change to retrieval is the most likely culprit. Inspecting the logged retrieved chunks per evaluation row reveals whether the new embedding model returns less relevant or contradictory passages, which the judge then flags as unsupported claims. This isolates retrieval as the cause before any remediation.

An automated judge that approves nearly every fluent summary is a biased measurement instrument, and the fix is to calibrate it rather than to change its sampling or the dataset size. Human-labeling a representative subset and measuring judge-human agreement exposes the leniency and tells the team how much to trust the scores. Only then can thresholds be set or a complementary fact-coverage check be added, producing a release signal that can be defended.

A complete RAG evaluation run needs metrics for both stages. Retrieval metrics score the chunks returned by the retriever, while generation metrics such as groundedness and relevance score the final answer against the question and context. Together they let the team pinpoint whether a quality issue originates in retrieval or in generation.

Context recall against a labeled ground-truth set directly tests whether the retriever brought back the documents that contain the answer. Because relevance measures topical similarity and can stay high even when authoritative chunks are missed, only recall can reveal that the rebuilt index now returns plausible but non-supporting passages, which would explain unsupported answers without any change in generation behavior.

Integrating MLflow Tracing provides granular visibility into the execution flow, including model inputs, outputs, and latency per step. This is critical for diagnosing complex chains where intermediate calls may be stalling. By instrumenting the code, engineers can identify which specific retrieval step or prompt generation block is contributing to the high latency, allowing for targeted optimization of the RAG pipeline components.

Groundedness evaluates whether the response's claims can be traced back to the retrieved context, which is precisely the gap identified in this review. Adding it to the MLflow LLM Evaluation run gives the team a metric that flags answers which sound correct but are not actually supported by the cited policy documents.

Effective monitoring requires both capturing data and evaluating that data against ground truth or automated metrics. Enabling inference tables provides the necessary raw data, while integrating with evaluation tools allows for systematic scoring of output quality. Together, these steps form the backbone of a robust monitoring strategy, ensuring that production drift is detected and quality is maintained according to business standards.

With context recall high, retrieval supplied the necessary facts, so the failure lies downstream in generation. Groundedness directly measures whether the answer's claims are supported by the retrieved context, so a low groundedness score localizes the problem to the generator ignoring or misreading context. That distinguishes a generation-faithfulness issue from a retrieval-ranking or labeling issue.

MLflow 3 production monitoring is designed for exactly this need: built-in scorers are attached to a deployed agent rather than invoked manually, and assessments are recorded on live traces inside the MLflow experiment. This removes custom scoring code, keeps quality signals tied to each request, and enables continuous monitoring without batch jobs or dashboard workarounds.

Inference tables capture the actual request and response payloads served by the endpoint, and Lakehouse Monitoring can build profiles and drift metrics over those tables. Together they provide the automated distributional comparison needed to detect when live questions diverge from the development baseline, which is exactly the monitoring goal.

Context recall and context precision are computed from the retrieved passages and ground-truth relevance, so they quantify what the retriever returned and in what order. Answer relevance, groundedness, and toxicity all depend on the generated response, which means they blend generation behavior into the score. Selecting the two context-based metrics keeps the harness focused on retrieval.

Mosaic AI Agent Evaluation supplies judges for safety and other quality dimensions, and it can be scheduled against inference table data so every production interaction is scored. Records that fail the safety criteria can be written to a separate table for review, giving the team continuous, automated monitoring rather than manual spot checks.

LLM drift occurs when the input data distribution or the nature of user queries changes over time, causing the model to perform worse than during validation. Because user expectations and language usage evolve, a model that performed well at launch may become less accurate as queries drift into new domains or change in tone. Monitoring this drift is essential for proactive maintenance and model updating.

The compliance requirement is per-answer attribution to a specific chunk, so the evaluation must verify each claim against the chunk it cites. Scoring the whole response against concatenated context allows a blended claim to pass whenever its components appear somewhere in the pool. Chunk-level citation assessment closes that gap by testing attribution directly, which is what distinguishes this failure from ordinary ungrounded hallucination.

Aggregate feedback rates are diluted by cohort mix, so an unchanged overall thumbs-down percentage can coexist with a severe regression inside one prompt template version, document source, or user cohort. Correlating the stored human feedback with MLflow traces on request_id and then slicing by those dimensions surfaces where the hallucination spike actually lives, which is the prerequisite for any targeted fix.

LLMs introduce unique challenges because their outputs are non-deterministic, high-dimensional, and often lack a single 'correct' answer. Traditional monitoring focuses on numerical drift and binary classification metrics like precision/recall. LLM evaluation requires semantic understanding, nuance assessment, and the ability to handle unstructured text, necessitating specialized tools like LLM-as-a-judge or human-in-the-loop validation to manage the subjectivity inherent in generative AI outputs.

Retrieval-stage quality is measured by whether the right documents are found and how many of the returned ones are relevant. Precision at k and recall against a curated question-to-document mapping capture those two dimensions directly, so regressions localize to indexing, chunking, or embeddings. Token count, time to first token, and toxicity all describe the generator or the serving layer, not retrieval.

Unity Catalog acts as the central governance layer for all Databricks data. By integrating evaluation with Unity Catalog, you ensure that all evaluation datasets, results, and models are discoverable, lineage-tracked, and governed. This is essential for compliance, ensuring that every model deployment is backed by a verifiable audit trail of its evaluation performance, which is a core requirement for enterprise AI deployments that must adhere to strict internal and external standards.

Reproducibility in evaluation requires logging not only the results but also the exact artifacts, environment, and code version used. MLflow's ability to log the model URI, the specific evaluation dataset version, and the code state (via git hash) ensures that an evaluation run can be perfectly recreated. This is mandatory for auditing and verifying improvements in LLM performance over time, preventing 'black box' results that cannot be validated.

Groundedness is the metric that inspects whether the answer's claims are supported by the retrieved context. Since the retriever is already delivering relevant chunks, a low groundedness score points squarely at the generator ignoring or misusing that evidence, giving the team a precise diagnostic for the stage that is failing.

Model quality decay in LLMs is best detected through continuous automated evaluation using a 'judge' model. By running a set of evaluation prompts through the production model and grading them against an LLM-as-a-judge, you can quantify performance trends. This proactive monitoring allows teams to identify when model behavior deviates from expected standards, triggering alerts before the issues impact a large number of end-users or require significant architectural changes.

Evaluating the retrieval stage requires assessing how well the system identifies relevant documents from the vector store before the LLM generates a response. Context Precision and Context Recall provide quantitative measures to ensure the retrieval engine is performing correctly. Without these metrics, it is impossible to determine if the LLM is failing due to poor generation or simply because it lacked the correct information to answer the query.

Inference tables are the native Databricks mechanism for capturing production traffic from a Model Serving endpoint into a governed Unity Catalog Delta table. They log payloads, responses, and metadata automatically, which is exactly what the team needs to analyze quality and latency over time. Other options either log only development activity or record lineage instead of runtime traffic, so they cannot satisfy the monitoring requirement.

MLflow Experiments act as the centralized repository for all model development and evaluation runs. By logging evaluation results to MLflow, teams create a permanent, version-controlled audit trail. This allows stakeholders to compare different iterations of models, review evaluation metrics, and verify that quality gates were met before deployment, ensuring compliance with internal AI governance policies and external regulatory frameworks for model transparency.

MLflow Model Evaluation provides a standardized framework to calculate metrics like RAGAS and perform side-by-side comparisons of model versions. By automating this process, organizations can ensure consistency across releases, track performance history over time, and reduce human bias. This systematic approach allows teams to make data-driven decisions when deciding whether to promote a model to production, ensuring reliability and auditability in the AI lifecycle.

Answering quality in RAG requires looking at two distinct dimensions: the correctness of the content (faithfulness) and the relevance of the answer to the user's intent. Faithfulness ensures the model remains grounded in the provided evidence, while answer relevance captures whether the model actually addressed the specific user question. These metrics provide a balanced view, ensuring the system is both accurate and useful, which is essential for end-user satisfaction.

A baseline dataset serves as a ground-truth reference point. By comparing the model's current outputs against this known high-quality set, engineers can objectively measure performance degradation or improvement. This benchmarking approach is fundamental to scientific model evaluation, ensuring that updates to the model or data retrieval logic do not silently introduce regressions in the quality of the generated answers during the deployment process.

Inference tables persist production request and response payloads as Delta, which makes them queryable by Lakehouse Monitoring. Building a monitor over that table, especially when joined with labeled data, yields scheduled quality and drift metrics that surface degradation automatically. This combination is the supported path for continuous, automated quality monitoring on Databricks rather than ad hoc manual inspection.

A robust RAG evaluation requires a holistic view that covers the entire pipeline. Evaluating just the generation or just the retrieval is insufficient because the quality of the final response is dependent on both stages. By measuring retrieval performance, generation faithfulness, and end-to-end answer quality, engineers obtain a full picture of where the application is succeeding and where it needs tuning to meet user requirements.

To verify that generated answers are faithful to the retrieved documents, the evaluation must compare the answer against the context. groundedness is the Agent Evaluation metric that performs this check, flagging responses that include unsupported claims. It is the appropriate pre-promotion quality gate for a RAG chain.

Score swings across identical runs are caused by nondeterministic judge sampling, so the fix must remove that randomness and make the decision rule stable. Setting the judge temperature to zero makes each judgment reproducible, and predefined thresholds turn noisy continuous scores into consistent pass/fail outcomes. Enlarging the judge, duplicating rows, or hiding per-row results leaves the stochasticity in place and keeps run-to-run comparisons unreliable.

A drop in relevance often suggests that the evaluation dataset (eval_set_v4) may have become outdated or misaligned with the current production use cases. If the user base or query patterns shift, the original evaluation set may no longer accurately reflect the actual performance requirements. Regularly refreshing the evaluation dataset to match current production trends is essential to ensure that metrics remain valid indicators of system performance.

When answers are factually correct but differ in how much supporting detail they include, a completeness metric is required. answer_completeness compares the response against the ground truth to check whether all key points are covered, making it the right built-in metric to expose the omission of supporting details that answer_correctness alone misses.

For a RAG pipeline, retrieval quality is assessed by relevance, which measures how well retrieved documents match the query. Generation quality is assessed by groundedness, which checks if the answer is supported by the retrieved context. Both are built-in metrics in MLflow LLM Evaluation and are specifically designed for RAG evaluation on Databricks.

Refusals rising while latency and token counts hold steady isolates the problem to retrieval quality rather than the model or the serving tier. Retrieval score distributions logged to the inference table show whether top-k chunks are becoming weaker matches, which is the earliest signal that the indexed corpus no longer covers current user questions. Checking capacity, token cost, or generation time cannot distinguish stale content from a healthy pipeline.

Identical inputs producing different scores across runs indicate the judge itself is nondeterministic, typically through model version drift or sampling temperature. Pinning the judge model version and forcing temperature to zero makes scoring reproducible, so observed differences can be attributed to the application. The other options either change the system under test, destroy evaluation fidelity, or hide variance rather than eliminate it.

End-user thumbs votes are weak but real labels tied to specific production requests. Joined with traces and request IDs, they drive triage of failing interactions and curation of new offline evaluation examples, connecting production monitoring to the evaluation dataset. They do not replace datasets, train models automatically, or improve metrics on their own.