Evaluation and Monitoring practice questions

Utilize traditional scikit-learn classification metrics such as F1-score and accuracy computed over raw token probabilities.

Why it fails: F1-score and accuracy over token probabilities measure classification correctness, not whether generated text is factually grounded in retrieved context. It is tempting because scikit-learn metrics are familiar and cheap, but hallucination and groundedness require an LLM judge comparing responses against source documents.

Deploy the model directly to a Databricks Model Serving endpoint and manually review a static sample of fifty chat logs.

Why it fails: Manual review of fifty static chat logs cannot compute MLflow's hallucination or groundedness judges at scale, and Model Serving deployment alone provides no scoring. It is tempting because serving endpoints expose the model for querying, which suits interactive testing, but the stem requires MLflow's automated LLM-judge evaluation utilities.

Compute standard BLEU and ROUGE scores using standard python packages without utilizing MLflow tracking capabilities.

Why it fails: BLEU and ROUGE rely strictly on exact token n-gram overlap against reference texts. They frequently fail to capture semantic equivalence, context adherence, and factual hallucination in advanced generative language applications where phrasing can vary wildly.

Relevance

Why it fails: Answer relevance measures how directly the generated response addresses the user input query, typically requiring only the query and the model response, without needing external retrieved document contexts or manual ground truth datasets.

Groundedness

Why it fails: Groundedness measures whether the generated response is derived exclusively from the retrieved context passages, requiring the model output and context chunks, but it does not strictly mandate the presence of an independent ground truth.

Toxicness

Why it fails: Toxicness evaluates the presence of hateful, profane, or abusive language within the generated text output. It operates solely on the generated response string and does not require retrieved contexts or ground truth references.

The inference table schema is incorrectly defined.

Why it fails: The exhibit shows a valid configuration for the catalog, schema, and table name. The issue is not the schema definition itself, but rather the explicit toggle that enables the platform's native monitoring services to process these logs into a dashboard format.

The endpoint does not support monitoring for chat-based LLMs.

Why it fails: Databricks Model Serving supports monitoring for all deployed models, including generative chat models. The limitation is not a platform-level constraint on chat architectures but a specific configuration setting within the endpoint definition that was explicitly disabled by the user during deployment.

The inference table is missing write permissions.

Why it fails: While permissions could potentially cause issues, the configuration itself shows valid identifiers. If permissions were the issue, the logs would not exist at all. The presence of the logs implies the configuration is correct, pointing to the explicit monitoring toggle as the primary failure point.

The judge model will always output the same score.

Why it fails: While a judge model might show bias, it does not necessarily produce static, identical scores. It may still assign varying grades, but those grades will be consistently skewed or logically inconsistent, which is a much more insidious problem than simple static output, as it masquerades as valid data.

The cost of evaluation will be zero.

Why it fails: Running LLM-as-a-judge incurs significant computational costs because the judge itself is an LLM that must be processed for every single response being evaluated. The cost is definitely not zero; in fact, it is often a significant portion of the total inference overhead for a production RAG system.

The judge model requires a separate vector database.

Why it fails: An LLM-as-a-judge does not necessarily require its own vector database. It simply processes the prompt and the response to generate a score or critique. The complexity arises from prompt engineering the judge and ensuring its reasoning logic is robust, not from additional data infrastructure requirements.

The model is hallucinating frequently.

Why it fails: The faithfulness score of 0.95 is very high, suggesting the model is staying strictly within the provided context. Hallucinations usually correlate with low faithfulness, so the data contradicts the conclusion that the model is failing to ground its answers in the provided knowledge base documents.

The generative model is broken and must be replaced.

Why it fails: The generative model is performing well at its specific job of being faithful to its input. Replacing the model would not solve the relevance issue, which is a symptom of the data retrieval stage. A better approach is to optimize the search query and the document ranking algorithm.

The evaluation dataset is biased.

Why it fails: While possible, it is premature to conclude dataset bias. The metrics provide a specific, actionable insight into the RAG pipeline's internal architecture: the retriever is failing to capture relevant information, even though the generator is working correctly. This is a standard architectural bottleneck in RAG, not a data bias issue.

Monitor the model's GPU temperature.

Why it fails: GPU temperature is a physical hardware metric that has no relevance to the safety or security of the model's responses. Auditors are interested in how the model handles sensitive data and potential attacks, not in the physical operating conditions of the server infrastructure hosting the application.

Increase the number of fine-tuning epochs.

Why it fails: Fine-tuning is a training strategy, not a security control. Increasing the number of epochs does not improve the security of the model against attacks or ensure that it follows safety guidelines. In fact, it might even increase the risk of overfitting to dangerous patterns in the training data.

Increase the number of retrieved chunks to ensure more information is included.

Why it fails: Increasing the number of chunks usually exacerbates context precision issues by introducing more irrelevant noise, making it harder for the LLM to focus on the key information, potentially lowering the faithfulness and relevance metrics further.

Fine-tune the generative model on the domain-specific dataset.

Why it fails: Fine-tuning improves style and adherence but rarely fixes retrieval precision issues. If the model is already providing faithful answers, the core issue is the quality of the provided context, not the model's ability to process it.

Reduce the temperature setting of the LLM to decrease hallucinations.

Why it fails: The faithfulness score of 0.91 is quite high, suggesting the model is already grounded in its context. Decreasing temperature might increase consistency but does not resolve the root cause of low retrieval precision identified.

Read the Delta table that stores the evaluation results and compute a mean over the per-row correctness column to derive a run-level score.

Why it fails: The results table holds per-row assessments, not the aggregated run-level metrics, and averaging a per-row judge column can diverge from how the framework aggregates. While useful for error analysis, this recomputation is fragile and does not give the job a stable, framework-defined metric to compare against the baseline.

Re-run mlflow.evaluate() with the same evaluation dataset and inspect the printed summary in the notebook output to confirm the metrics match the baseline.

Why it fails: Re-running evaluation recreates judge calls and does not persist a comparison against the earlier baseline; the printed notebook summary is not machine-readable inside the job. It also costs additional judge invocations and can drift between runs, so it cannot serve as a reliable programmatic gate for the next deployment.

Enable inference tables on the serving endpoint and compare the latest logged request payloads to the baseline metric values stored in the model registry.

Why it fails: Inference tables capture live serving traffic, not offline evaluation runs, and the model registry stores model versions and tags rather than evaluation metric values. Comparing payloads to registry entries mixes two unrelated data sources, so the job would have no consistent numeric baseline to gate the deployment on.

Version the evaluation dataset in Unity Catalog and pin the version used by each evaluation run.

Why it fails: Versioning is good practice for reproducibility, but it does not by itself provide ground truth or relevance labels. The question asks for choices that enable retrieval and answer quality judgments, and versioning alone supplies neither. It supports comparability of the artifact, not the semantic content required by judges.

Limit the dataset to questions the current agent already answers correctly to keep scores stable.

Why it fails: Selecting only easy, already-correct questions creates a ceiling effect and hides regressions. A useful evaluation set deliberately includes challenging and edge-case questions. This choice produces misleadingly high and stable scores that cannot detect quality degradation when the agent changes.

Store the dataset only as raw prompt strings so the judge can infer the intended answer.

Why it fails: Raw prompts without references force the judge to guess the intended answer, producing unstable and unverifiable scores. Regression suites need fixed expectations to compare versions. This choice undermines both retrieval and answer quality measurement and makes historical runs non-comparable.

Answer relevance

Why it fails: Answer relevance measures how well the generated response addresses the user's question, not whether every claim is supported by the retrieved context. In this scenario, an answer could be perfectly on-topic about interest rates while still fabricating a rate value, so relevance would score high and hide the grounding failure. The engineer needs a metric tied to the retrieved context specifically.

Latency measured at the serving endpoint

Why it fails: Endpoint latency describes how long the request took and says nothing about whether the answer's numbers are supported by retrieved policy documents. A fast but hallucinated interest rate would still pass a latency check, so this metric cannot quantify the grounding failure described. Latency monitoring belongs to performance observability, not answer-faithfulness evaluation.

Token usage per request

Why it fails: Token usage tracks prompt and completion size for cost and context-window management. It cannot tell whether an invented interest rate appears in the retrieved chunks, because a short answer can still be unfaithful. This metric would only help the engineer reason about prompt trimming or cost, not about whether the answer is grounded in the policy documents.

Databricks SQL Alerts

Why it fails: SQL Alerts are designed for monitoring table state changes or specific query results rather than capturing high-throughput inference payloads. They lack the native integration with Model Serving endpoints required to stream request-response pairs effectively into a structured format for large-scale LLM evaluation and performance observability.

Delta Live Tables Audit Logs

Why it fails: Delta Live Tables audit logs track pipeline execution status and lineage, not the internal payload of a served model. While they are useful for operational monitoring of ETL pipelines, they do not provide the granular request-response visibility necessary for evaluating generative AI model behavior and output quality.

MLflow Experiment Tracking

Why it fails: MLflow Experiment Tracking is primarily for logging metrics and parameters during the development and training phases of a model. It is not designed to capture live production inference data in real-time. Using it for production monitoring would require custom instrumentation that bypasses the native efficiency of inference tables.

Model Training Loss

Why it fails: Training loss measures how well the model has converged on its training dataset during the optimization phase. It provides no information about the quality of retrieval or the relevance of context fetched from an external vector database at inference time during production user interactions.

Inference Latency

Why it fails: Inference latency tracks the time taken to generate a response but ignores the quality or accuracy of that response. While essential for operational performance, latency does not provide any qualitative insight into whether the system retrieved the correct information to satisfy the user request.

Parameter Count

Why it fails: Parameter count is a static attribute of the model architecture, not a dynamic performance or quality metric. It does not fluctuate during inference and cannot be used to monitor the effectiveness of a RAG pipeline's retrieval mechanism or the accuracy of generated responses.

Set up an automatic retraining job on the endpoint.

Why it fails: Automatic retraining of large language models is rarely performed directly on a serving endpoint. LLMs require significant compute and complex fine-tuning processes outside the scope of simple model serving. Monitoring should focus on evaluation and RAG optimization rather than automated model re-training of the base LLM.

Increase the GPU count to maximize inference throughput.

Why it fails: Scaling compute resources impacts operational latency and throughput, but it does not contribute to the evaluation or monitoring of model quality. While performance is a component of monitoring, the question specifically targets the evaluation and oversight of model behavior and output quality in production.

Delete all request logs after 24 hours to save storage.

Why it fails: Deleting logs immediately destroys the historical record needed for long-term trend analysis and model evaluation. Monitoring requires a longitudinal view to detect subtle shifts in model behavior or data drift, making the retention of inference logs a critical requirement for any production AI system.

To increase the number of tokens the model can generate.

Why it fails: The baseline dataset is used for evaluation and comparison, not for training or extending the model's generative capacity. Token limits are defined by the model architecture and configuration, not by the evaluation data provided during a monitoring or assessment task.

To act as a secondary training set for fine-tuning.

Why it fails: Evaluation datasets are strictly for testing and assessment; they should never be confused with training or fine-tuning data. Using evaluation data for training causes data leakage, which invalidates the metrics and leads to overly optimistic performance assessments that do not reflect real-world user interaction.

To optimize the vector database search latency.

Why it fails: Vector search optimization requires index tuning and hardware configuration, not an evaluation dataset. While the evaluation dataset can measure retrieval accuracy, it does not directly optimize the underlying search speed or index performance of the database itself.

Lack of high-throughput API endpoints.

Why it fails: LLM services on Databricks are designed for high-throughput and low-latency, mirroring traditional API capabilities. The challenge in LLM monitoring is the nature of the data and the assessment of quality, not the ability of the underlying infrastructure to handle high request volumes and concurrent connections.

The model's inability to connect to internet data.

Why it fails: While some models are restricted, this is a model capability feature rather than a specific monitoring challenge. Most enterprise RAG architectures explicitly use internal, private data. The monitoring challenge remains the quality of the generation, regardless of whether the data source is the internet or internal.

Context Precision

Why it fails: Context precision measures the quality of the retrieved chunks relative to the user query, focusing on whether the retriever found relevant info. It does not measure the generative model's behavior or its ability to stick to the provided context when formulating an answer to the user.

Retrieval Recall

Why it fails: Retrieval recall measures the proportion of relevant information successfully retrieved from the knowledge base. While important for RAG, it does not assess how the generative model uses that information. Even with perfect recall, the generative model could still choose to ignore the context and hallucinate.

Semantic Similarity

Why it fails: Semantic similarity measures how close the generated answer is to a reference ground truth. It is a coarse-grained metric that does not specifically reveal whether the model ignored the context. An answer could be semantically similar to a reference but still hallucinated, failing the faithfulness requirement.

Only log the final scalar metrics.

Why it fails: Logging only scalar metrics discards the context needed for reproducibility, such as the data used, the prompt template, and the model version. Without this metadata, it is impossible to re-run the evaluation to confirm results or debug discrepancies, rendering the monitoring process non-reproducible and technically opaque.

Hardcode the dataset paths in the notebook.

Why it fails: Hardcoding paths is a major anti-pattern that breaks reproducibility as data evolves or moves. It makes the evaluation run dependent on a specific, mutable environment state rather than a versioned artifact, preventing consistent benchmarking across different development, testing, and production stages of the model lifecycle.

Use a global variable for model versioning.

Why it fails: Global variables are unreliable and do not persist across sessions, making them unsuitable for maintaining a history of evaluation runs. MLflow tracking provides a robust, centralized metadata store that is specifically designed to handle versioning and provenance, which global variables simply cannot replicate in a professional environment.

Inference Latency

Why it fails: Inference latency is a performance metric, not a quality metric. While it is important for the operational health of the application, it does not inform you about the factual accuracy, relevance, or grammatical correctness of the content produced by the generative model during the interaction.

GPU Utilization

Why it fails: GPU utilization is a hardware-level monitoring metric used for infrastructure scaling and cost optimization. It provides zero insight into the quality of the generative AI outputs. Focusing on utilization while ignoring output quality is a common pitfall that overlooks the core objective of the AI application's success.

Training Iterations

Why it fails: Training iterations are a parameter of the fine-tuning process. They reflect the effort spent during development but tell you nothing about the quality of the resulting model's outputs in a production RAG environment. Monitoring must be based on inference-time results, not internal training process statistics.

The model's weights change during inference.

Why it fails: Model weights are static during inference in a production serving endpoint. Drift refers to the shift in input data or the changing quality of outputs relative to the environment, not a physical change in the model's neural network parameters or architecture during the serving phase.

The model generating the same answer too often.

Why it fails: This describes a specific failure mode often called 'repetitiveness' or 'low diversity.' While this is a quality problem, it is not what is defined as 'drift' in machine learning monitoring. Drift refers to a statistical shift in the underlying data distribution, not the specific diversity of outputs.

Increased latency in the serving endpoint.

Why it fails: Latency is a performance metric related to infrastructure throughput and compute efficiency. It is not drift. Drift specifically refers to a degradation in the quality or predictive accuracy of the model's output resulting from a change in the input data distribution, not the processing speed of the service.

To increase the model's training speed.

Why it fails: Unity Catalog is a governance and data management tool; it has no impact on model training speed or computational efficiency. Integration provides lineage, versioning, and access control, which are governance features, not performance-enhancing features for the model's internal optimization or training process.

To automatically delete old evaluation data.

Why it fails: Unity Catalog is designed to keep data available, secure, and traceable for long-term audit purposes. It is not an automated data-cleanup tool. Automatic deletion of evaluation records would actually violate the goals of a strong governance framework, which requires data retention for historical compliance and audit review.

To bypass the need for prompt engineering.

Why it fails: Prompt engineering is a critical task for LLM success that exists independently of governance tools. Unity Catalog provides the framework to manage the results of your prompt engineering, but it does not perform the engineering itself, nor does it make the skill any less necessary for high-quality outcomes.