Databricks-GenAI-Assoc Evaluation and Monitoring Practice Test 2 — 15 Questions

Question 1

When monitoring a RAG application, you notice a high discrepancy between the retrieved context and the generated answer. Which metric would specifically help identify if the model is ignoring the provided context?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Faithfulness. Faithfulness measures whether the generated answer is derived exclusively from the retrieved context. If a model generates information not supported by the source text, it is 'hallucinating' or ignoring the context. Monitoring faithfulness is crucial for RAG systems because it directly detects when the model drifts away from the ground truth provided by the internal knowledge base, which is the primary value proposition of a RAG architecture.

Answer

Context Precision

Answer

Retrieval Recall

Answer

Semantic Similarity

Question 2

A team is using MLflow to track their model evaluation runs. They want to ensure that every evaluation run is reproducible. What is the best practice to achieve this in Databricks?

Accepted Answer

Log the model URI, dataset version, and code hash.. Reproducibility in evaluation requires logging not only the results but also the exact artifacts, environment, and code version used. MLflow's ability to log the model URI, the specific evaluation dataset version, and the code state (via git hash) ensures that an evaluation run can be perfectly recreated. This is mandatory for auditing and verifying improvements in LLM performance over time, preventing 'black box' results that cannot be validated.

Answer

Only log the final scalar metrics.

Answer

Hardcode the dataset paths in the notebook.

Answer

Use a global variable for model versioning.

Question 3

When evaluating LLM outputs, which TWO metrics are most appropriate for measuring the 'quality' of a response in a RAG system? (Choose two)

Accepted Answer

Faithfulness. Answering quality in RAG requires looking at two distinct dimensions: the correctness of the content (faithfulness) and the relevance of the answer to the user's intent. Faithfulness ensures the model remains grounded in the provided evidence, while answer relevance captures whether the model actually addressed the specific user question. These metrics provide a balanced view, ensuring the system is both accurate and useful, which is essential for end-user satisfaction.

Answer

Inference Latency

Answer

GPU Utilization

Answer

Training Iterations

Question 4

Which of the following describes the 'drift' phenomenon in the context of LLM monitoring?

Accepted Answer

A shift in user query patterns over time.. LLM drift occurs when the input data distribution or the nature of user queries changes over time, causing the model to perform worse than during validation. Because user expectations and language usage evolve, a model that performed well at launch may become less accurate as queries drift into new domains or change in tone. Monitoring this drift is essential for proactive maintenance and model updating.

Answer

The model's weights change during inference.

Answer

The model generating the same answer too often.

Answer

Increased latency in the serving endpoint.

Question 5

Why should you integrate Mosaic AI Model Evaluation with Unity Catalog?

Accepted Answer

To provide centralized governance and lineage.. Unity Catalog acts as the central governance layer for all Databricks data. By integrating evaluation with Unity Catalog, you ensure that all evaluation datasets, results, and models are discoverable, lineage-tracked, and governed. This is essential for compliance, ensuring that every model deployment is backed by a verifiable audit trail of its evaluation performance, which is a core requirement for enterprise AI deployments that must adhere to strict internal and external standards.

Answer

To increase the model's training speed.

Answer

To automatically delete old evaluation data.

Answer

To bypass the need for prompt engineering.

Question 6

An enterprise LLM application on Databricks is experiencing high latency and inconsistent responses. Which approach best enables observability to identify the root cause of these performance bottlenecks within the LLM pipeline?

Accepted Answer

Use MLflow Tracing to record inputs, outputs, and execution duration of each step in the chain.. Integrating MLflow Tracing provides granular visibility into the execution flow, including model inputs, outputs, and latency per step. This is critical for diagnosing complex chains where intermediate calls may be stalling. By instrumenting the code, engineers can identify which specific retrieval step or prompt generation block is contributing to the high latency, allowing for targeted optimization of the RAG pipeline components.

Answer

Enable standard Spark UI logs to monitor cluster resource utilization during inference.

Answer

Review the Databricks SQL query history to identify slow retrieval operations.

Answer

Deploy an external monitoring agent to scan the model server network traffic.

Question 7

Which TWO evaluation approaches are most effective for measuring the quality of a RAG pipeline's retrieval stage?

Accepted Answer

Context Precision. Evaluating the retrieval stage requires assessing how well the system identifies relevant documents from the vector store before the LLM generates a response. Context Precision and Context Recall provide quantitative measures to ensure the retrieval engine is performing correctly. Without these metrics, it is impossible to determine if the LLM is failing due to poor generation or simply because it lacked the correct information to answer the query.

Answer

Model Answer Faithfulness

Answer

Answer Semantic Similarity

Answer

LLM Latency Monitoring

Question 8

What is the primary benefit of using MLflow Model Evaluation for generative AI applications compared to manual evaluation methods?

Accepted Answer

It enables consistent, repeatable evaluation metrics across different model versions.. MLflow Model Evaluation provides a standardized framework to calculate metrics like RAGAS and perform side-by-side comparisons of model versions. By automating this process, organizations can ensure consistency across releases, track performance history over time, and reduce human bias. This systematic approach allows teams to make data-driven decisions when deciding whether to promote a model to production, ensuring reliability and auditability in the AI lifecycle.

Answer

It automatically rewrites the prompt templates based on user feedback.

Answer

It removes the need for ground truth datasets during the testing phase.

Answer

It creates the vector index automatically from raw unstructured data.

Question 9

When monitoring a production LLM, you detect a drift where the model's responses are becoming increasingly verbose and less helpful compared to the baseline. Which strategy is most effective for detecting this quality decay?

Accepted Answer

Implement automated LLM-as-a-judge evaluations on a sample of production inputs.. Model quality decay in LLMs is best detected through continuous automated evaluation using a 'judge' model. By running a set of evaluation prompts through the production model and grading them against an LLM-as-a-judge, you can quantify performance trends. This proactive monitoring allows teams to identify when model behavior deviates from expected standards, triggering alerts before the issues impact a large number of end-users or require significant architectural changes.

Answer

Monitor system-level CPU and memory usage of the inference cluster.

Answer

Set up alerts for high request volume spikes in the API logs.

Answer

Require manual reviews for every single response generated by the model.

Question 10

Which THREE components are critical to include in a comprehensive evaluation strategy for a RAG-based Generative AI application?

Accepted Answer

Retrieval metrics (e.g., Hit Rate, MRR). A robust RAG evaluation requires a holistic view that covers the entire pipeline. Evaluating just the generation or just the retrieval is insufficient because the quality of the final response is dependent on both stages. By measuring retrieval performance, generation faithfulness, and end-to-end answer quality, engineers obtain a full picture of where the application is succeeding and where it needs tuning to meet user requirements.

Answer

Hardware temperature monitoring for GPUs

Answer

Network packet loss percentage

Question 11

Your organization requires an auditable record of all LLM evaluation results for compliance. Which Databricks feature provides the best centralized storage for these evaluation runs?

Accepted Answer

MLflow Experiments. MLflow Experiments act as the centralized repository for all model development and evaluation runs. By logging evaluation results to MLflow, teams create a permanent, version-controlled audit trail. This allows stakeholders to compare different iterations of models, review evaluation metrics, and verify that quality gates were met before deployment, ensuring compliance with internal AI governance policies and external regulatory frameworks for model transparency.

Answer

Databricks File System (DBFS) text logs

Answer

Unity Catalog volume for raw CSV storage

Answer

Git commit history of the inference code

Question 12

Refer to the exhibit. An engineer observes an unexpected drop in "relevance" for the latest deployment. What is the most likely cause related to the evaluation process itself?

Accepted Answer

The evaluation dataset is misaligned with current production query trends.. A drop in relevance often suggests that the evaluation dataset (eval_set_v4) may have become outdated or misaligned with the current production use cases. If the user base or query patterns shift, the original evaluation set may no longer accurately reflect the actual performance requirements. Regularly refreshing the evaluation dataset to match current production trends is essential to ensure that metrics remain valid indicators of system performance.

Answer

The model was trained on too much data.

Answer

The inference cluster has too much compute capacity.

Answer

The faithfulness score increased too much.

Question 13

A Databricks Generative AI engineer has deployed a RAG application and is now setting up production monitoring. They want to automatically detect when the distribution of incoming user questions diverges from the distribution seen during development, so they can trigger retraining or prompt adjustments. Which Databricks capability should they configure?

Accepted Answer

Inference tables with Lakehouse Monitoring for the endpoint's payload and response columns. Inference tables capture the actual request and response payloads served by the endpoint, and Lakehouse Monitoring can build profiles and drift metrics over those tables. Together they provide the automated distributional comparison needed to detect when live questions diverge from the development baseline, which is exactly the monitoring goal.

Answer

Mosaic AI Agent Evaluation with a custom LLM judge for every request

Answer

Delta Live Tables expectations on the raw question stream

Answer

MLflow Model Registry stage transitions with webhook notifications

Question 14

A team is using MLflow LLM Evaluation with the built-in answer_correctness metric to compare two prompt templates for a question-answering application. They notice that answer_correctness scores are nearly identical, but manual review shows one template produces answers that are factually correct yet omit key supporting details. Which additional built-in metric should they add to their evaluation to surface this difference?

Accepted Answer

answer_completeness. When answers are factually correct but differ in how much supporting detail they include, a completeness metric is required. answer_completeness compares the response against the ground truth to check whether all key points are covered, making it the right built-in metric to expose the omission of supporting details that answer_correctness alone misses.

Answer

groundedness

Answer

toxicity

Answer

answer_similarity

Question 15

A Generative AI engineer is deploying a new version of a RAG chain to a Mosaic AI Model Serving endpoint. Before promoting it to production, they want to run an evaluation that checks whether the generated answers are faithful to the retrieved documents. Which Mosaic AI Agent Evaluation metric should they examine?

Accepted Answer

groundedness. To verify that generated answers are faithful to the retrieved documents, the evaluation must compare the answer against the context. groundedness is the Agent Evaluation metric that performs this check, flagging responses that include unsupported claims. It is the appropriate pre-promotion quality gate for a RAG chain.

Answer

chunk_relevance

Answer

latency

Answer

request_count