Databricks-GenAI-Assoc Evaluation and Monitoring — 10 Questions

Question 1

An enterprise team is preparing to deploy a custom Large Language Model for a customer-facing support application on Databricks. They need to evaluate the model's responses for hallucination and groundedness using MLflow. Which specific evaluation utility should the ML Engineer utilize?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Invoke mlflow.evaluate() incorporating LLM-as-a-judge metrics to assess retrieved context alignment and factual consistency.. MLflow provides specialized GenAI evaluation functions, notably mlflow.evaluate(), which natively integrates LLM-judges for metrics like groundedness and hallucination using context passages. This is crucial for production deployments to ensure reliability before user exposure. Ensuring systematic metric tracking prevents deployment regressions.

Answer

Utilize traditional scikit-learn classification metrics such as F1-score and accuracy computed over raw token probabilities.

Answer

Deploy the model directly to a Databricks Model Serving endpoint and manually review a static sample of fifty chat logs.

Answer

Compute standard BLEU and ROUGE scores using standard python packages without utilizing MLflow tracking capabilities.

Question 2

An AI engineer is configuring MLflow LLM Evaluation on Databricks to assess a Retrieval-Augmented Generation (RAG) pipeline. Which TWO evaluation metrics require both the retrieved context and the ground truth answer to compute accurately? (Select TWO)

Accepted Answer

Retrieval relevance. Evaluating RAG pipelines requires distinguishing between retrieval quality and generation quality. Certain metrics rely specifically on the retrieved context chunks, whereas others evaluate answer relevance against the provided ground truth or user query. Selecting the right combination ensures comprehensive diagnostic visibility.

Answer

Relevance

Answer

Groundedness

Answer

Toxicness

Question 3

Refer to the exhibit. The engineer notices that while inference logs are being captured, the standard Databricks monitoring dashboard for the endpoint is not populating any quality metrics. What is the most likely cause?

Accepted Answer

The `monitoring_enabled` flag is set to false.. The JSON configuration indicates that while `inference_tables` are enabled, the `monitoring_enabled` flag is set to false. Databricks Model Serving requires explicit configuration to bridge inference tables with automated monitoring dashboards. Without this flag, the platform does not trigger the internal evaluation processes that calculate and display quality metrics such as drift scores or response accuracy trends within the dashboard interface.

Answer

The inference table schema is incorrectly defined.

Answer

The endpoint does not support monitoring for chat-based LLMs.

Answer

The inference table is missing write permissions.

Question 4

An organization is using 'LLM-as-a-judge' to evaluate their RAG application. What is the primary risk associated with this approach if the judge model is not carefully selected?

Accepted Answer

The evaluation process will become too fast.. Using an LLM to evaluate another LLM introduces bias and potential self-reinforcement loops, especially if the judge model is less capable or shares the same training biases as the model being evaluated. If the judge model lacks the required reasoning capability, it may systematically misgrade outputs. This creates a false sense of security where metrics look good but user experience remains poor due to flawed evaluation logic.

Answer

The judge model will always output the same score.

Answer

The cost of evaluation will be zero.

Answer

The judge model requires a separate vector database.

Question 5

Refer to the exhibit. An engineer evaluates a RAG pipeline and sees the provided results. What is the most reasonable conclusion regarding the system's performance?

Accepted Answer

The retrieval component needs investigation.. The evaluation shows high faithfulness (0.95) but low answer relevance (0.60). This indicates the model is very good at using the retrieved information correctly but is struggling to align that information with the actual user query. The retriever might be returning accurate, high-quality documents that are simply not relevant to the user's question. The engineer should investigate the retrieval logic or the embedding strategy, not the generative model itself.

Answer

The model is hallucinating frequently.

Answer

The generative model is broken and must be replaced.

Answer

The evaluation dataset is biased.

Question 6

A security auditor requires proof that the LLM application is safe for production. Which THREE of the following monitoring/evaluation tasks would satisfy this requirement? (Choose three)

Accepted Answer

Evaluate the model for compliance with system prompt instructions.. Security and safety in LLMs require a multi-faceted approach involving data grounding, prompt injection detection, and content filtering. Ensuring the model follows guardrails prevents malicious or off-topic usage, while monitoring for groundedness prevents misinformation. By combining these, you create a robust defense that demonstrates the system behaves predictably and securely under various adversarial conditions, which is exactly what a security auditor looks for in a production-ready AI implementation.

Answer

Monitor the model's GPU temperature.

Answer

Increase the number of fine-tuning epochs.

Question 7

Refer to the exhibit. An LLM-powered RAG system returns these metrics. Which action should the team prioritize to improve performance?

Accepted Answer

Implement a re-ranking mechanism to filter irrelevant documents before generation.. The low context_precision score of 0.45 indicates that the retrieval component is likely fetching irrelevant documents, leading to noise in the prompt context. By focusing on improving the retrieval mechanism, such as implementing hybrid search, re-ranking models, or optimizing chunking strategies, the system will provide higher quality data to the LLM, directly improving the precision of the generated answers without needing to retrain the model itself.

Answer

Increase the number of retrieved chunks to ensure more information is included.

Answer

Fine-tune the generative model on the domain-specific dataset.

Answer

Reduce the temperature setting of the LLM to decrease hallucinations.

Question 8

A GenAI engineer runs mlflow.evaluate() with model_type="databricks-agent" on a RAG chain registered as a Unity Catalog function. The run completes and the evaluation results table is written, but the engineer needs to compare this run against the previous production baseline and programmatically gate the next deployment on it inside a Databricks job. Which action should the engineer take?

Accepted Answer

Query the MLflow run's metrics with mlflow.search_runs() filtering on the run ID and compare the returned metric values to the baseline thresholds in the job.. MLflow LLM Evaluation records aggregate judge metrics on the evaluation run itself, so pulling them back with mlflow.search_runs() gives the job a stable numeric signal to compare against a stored baseline. Re-running evaluation, recomputing from per-row results, or mixing in serving telemetry all fail to provide that consistent, framework-defined run-level metric needed for an automated promotion gate.

Answer

Read the Delta table that stores the evaluation results and compute a mean over the per-row correctness column to derive a run-level score.

Answer

Re-run mlflow.evaluate() with the same evaluation dataset and inspect the printed summary in the notebook output to confirm the metrics match the baseline.

Answer

Enable inference tables on the serving endpoint and compare the latest logged request payloads to the baseline metric values stored in the model registry.

Question 9

A team is building an offline regression suite for a RAG agent using MLflow LLM Evaluation on Databricks. They want the evaluation dataset to support both retrieval quality and answer quality judgments and to make future runs comparable. Which TWO dataset design choices should they make? (Choose two.)

Accepted Answer

Include the expected retrieved document identifiers or relevant context for each question.. Ground-truth expected responses enable answer correctness and similarity scoring, while expected document identifiers or relevant context enable retrieval metrics such as context recall and precision. Together they let one dataset drive both stages of RAG evaluation and keep runs comparable over time as the agent and retriever evolve.

Answer

Version the evaluation dataset in Unity Catalog and pin the version used by each evaluation run.

Answer

Limit the dataset to questions the current agent already answers correctly to keep scores stable.

Answer

Store the dataset only as raw prompt strings so the judge can infer the intended answer.

Question 10

A GenAI engineer at a retail bank has deployed a RAG assistant on Mosaic AI Model Serving. Compliance requires that any answer about account balances must be grounded strictly in the retrieved policy documents. During evaluation, the engineer notices that some answers include invented interest rates not present in any retrieved chunk. Which metric should the engineer inspect FIRST to quantify this grounding failure?

Accepted Answer

Context sufficiency. The failure is that generated answers contain facts absent from the retrieved chunks. Groundedness-style metrics such as context sufficiency evaluate whether the retrieved context supports the generated claims, which is exactly the compliance requirement here. Answer relevance, latency, and token usage measure different properties and would not expose unsupported interest-rate values.

Answer

Answer relevance

Answer

Latency measured at the serving endpoint

Answer

Token usage per request