Databricks-GenAI-Assoc Evaluation and Monitoring Practice Test 3 — 15 Questions

Question 1

A GenAI engineering team has deployed a customer-support RAG chain on Databricks and registered it in Unity Catalog. They now want MLflow 3 to automatically score every production request for groundedness and relevance without writing custom scoring code, and to persist those assessments against the logged traces. Which approach should they use?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Attach MLflow scorers to the deployed agent so that assessments are computed on live traces and stored in the MLflow experiment.. MLflow 3 production monitoring is designed for exactly this need: built-in scorers are attached to a deployed agent rather than invoked manually, and assessments are recorded on live traces inside the MLflow experiment. This removes custom scoring code, keeps quality signals tied to each request, and enables continuous monitoring without batch jobs or dashboard workarounds.

Answer

Configure the endpoint's autoscaling policy to collect evaluation metrics alongside throughput metrics.

Answer

Create a Databricks SQL dashboard that re-runs mlflow.evaluate() on the inference table every five minutes.

Answer

Enable the system table system.serving.endpoint_usage and query it for groundedness scores.

Question 2

A team runs an LLM-as-a-judge evaluation on Databricks using the built-in `mlflow.evaluate()` with `model_type="databricks-agent"`. They notice that the judge model, a serving endpoint, is producing scores that are consistently inflated compared to human review. Which configuration change should the team make first to improve the reliability of the evaluation?

Accepted Answer

Switch the judge to a different, stronger model endpoint and calibrate it using a small human-labeled dataset with known ground-truth scores.. LLM judges can exhibit systematic bias, such as consistently inflated scores. The most effective first step is to use a stronger judge model and calibrate it against human-labeled data with known scores. This improves reliability and aligns with Databricks recommendations for validating judges before using them in production monitoring. Simply changing temperature or filtering data does not address the root cause.

Answer

Reduce the number of evaluation examples to only those where the judge and human agree, then report the average score.

Answer

Increase the temperature parameter of the judge model to 1.0 to encourage more diverse scoring.

Answer

Replace the LLM judge with a BLEU score comparison against a reference answer for every question.

Question 3

A team runs a RAG application on Mosaic AI Model Serving and logs all requests to an inference table. Reviewers report that answers are sometimes fluent but contradict the retrieved documents. The team wants a recurring, automated check that quantifies this contradiction on production traffic and alerts when it exceeds a threshold. Which approach best fits?

Accepted Answer

Schedule a Databricks job that runs mlflow.evaluate() with the groundedness judge over recent inference-table records and writes results to a Delta table for alerting.. Groundedness evaluation compares generated answers against retrieved context, which is precisely the failure mode described. Running it as a scheduled job over recent inference-table records produces a quantified, recurring metric, and persisting results to Delta supports alerting on threshold breaches. Operational metrics and manual review cannot detect semantically unsupported but fluent responses.

Answer

Enable inference table payload logging and inspect raw prompt and response text manually each week.

Answer

Increase the number of retrieved chunks per query so the model has more context to draw from.

Answer

Monitor the endpoint's p95 latency and error rate in the serving endpoint metrics and alert when either degrades.

Question 4

A Generative AI engineer at a retail company has deployed a RAG chatbot on a Databricks Mosaic AI Model Serving endpoint. The team wants to capture per-request evaluation data, including the user's question, the retrieved context chunks, and the model's response, so they can analyze response quality over time. Which Databricks feature should they use to collect this data for downstream evaluation?

Accepted Answer

Inference tables. Inference tables are the Databricks mechanism that logs request and response payloads for Mosaic AI Model Serving endpoints. Because the logged payloads include the user question, retrieved context, and generated answer, the team gains the raw material needed to run MLflow LLM Evaluation or custom monitoring jobs later against real production traffic.

Answer

MLflow Model Registry

Answer

Delta Live Tables expectations

Answer

Unity Catalog lineage

Question 5

A team runs a RAG chatbot whose MLflow evaluation with the built-in groundedness judge previously scored well. After they swap the retriever for a new embedding model, groundedness scores drop sharply even though the generator model and prompt are unchanged. They confirm the judge model itself is unchanged. Which action should they take FIRST to diagnose the regression?

Accepted Answer

Inspect the retrieved context chunks logged per evaluation row to verify whether the new retriever is returning passages that no longer support the generated claims.. Groundedness measures whether the answer is supported by the retrieved context, so a change to retrieval is the most likely culprit. Inspecting the logged retrieved chunks per evaluation row reveals whether the new embedding model returns less relevant or contradictory passages, which the judge then flags as unsupported claims. This isolates retrieval as the cause before any remediation.

Answer

Roll back the embedding model change immediately and re-run the evaluation to confirm the previous score returns.

Answer

Retrain the judge model on a fresh labeled dataset so its scoring distribution aligns with the new embedding model.

Answer

Increase the temperature of the generator model to encourage more diverse phrasing that the groundedness judge can score more reliably.

Question 6

A Generative AI engineer is using MLflow Tracing to monitor a RAG application deployed on Mosaic AI Model Serving. They want to capture the retrieved documents, the final prompt, and the model's response for each request to debug a quality issue. Which approach should they use to ensure all three are logged in a single trace?

Accepted Answer

Instrument the application with the MLflow tracing SDK, creating spans for retrieval, prompt construction, and generation within the same trace context.. To capture retrieved documents, the final prompt, and the model response in a single trace, the engineer should use MLflow Tracing with manual instrumentation. Spans for each step share a trace context, providing a unified view. This is the native Databricks approach for debugging RAG applications and integrates with Mosaic AI Model Serving for production monitoring.

Answer

Use the `mlflow.evaluate()` API with a custom evaluator that logs retrieved documents, prompt, and response as metrics.

Answer

Enable inference table logging on the serving endpoint and query the payload column for the retrieved documents and response.

Answer

Configure the serving endpoint to log all requests to a Delta table and then join with a separate table of retrieved documents using a request ID.

Question 7

A financial services team runs a RAG assistant on Databricks that answers questions about internal policy documents. During a review, the team finds that for many questions the answer is factually correct but cites a document that does not actually contain the supporting statement. They want an MLflow LLM Evaluation metric that specifically detects when the response is not supported by the retrieved context. Which metric should they add to their evaluation run?

Accepted Answer

groundedness. Groundedness evaluates whether the response's claims can be traced back to the retrieved context, which is precisely the gap identified in this review. Adding it to the MLflow LLM Evaluation run gives the team a metric that flags answers which sound correct but are not actually supported by the cited policy documents.

Answer

answer_similarity

Answer

retrieval_precision

Answer

toxicity

Question 8

An engineer registers a GenAI agent to Unity Catalog and enables inference tables on its Mosaic AI Model Serving endpoint. They want to automatically detect when response quality degrades in production without waiting for human review. Which capability should they configure to achieve continuous automated quality monitoring?

Accepted Answer

A Databricks Lakehouse Monitoring monitor over the inference table, joined with a labeled evaluation dataset to compute quality metrics on a schedule.. Inference tables persist production request and response payloads as Delta, which makes them queryable by Lakehouse Monitoring. Building a monitor over that table, especially when joined with labeled data, yields scheduled quality and drift metrics that surface degradation automatically. This combination is the supported path for continuous, automated quality monitoring on Databricks rather than ad hoc manual inspection.

Answer

Configure autoscaling on the serving endpoint so additional model replicas absorb traffic spikes and stabilize response quality.

Answer

Set up a scheduled job that calls the endpoint with synthetic prompts and records latency percentiles to a Delta table for trending.

Answer

Enable model signature enforcement on the endpoint so requests that violate the expected schema are rejected before scoring.

Question 9

A team has deployed a RAG chatbot on Databricks and enabled inference table logging. They want to set up automated monitoring to detect when the average response length increases significantly compared to the baseline. Which Databricks feature should they use to create this monitor?

Accepted Answer

Lakehouse Monitoring for the inference table, with a custom metric for response length.. Lakehouse Monitoring is designed to monitor Delta tables, including inference tables, for data quality and drift. By creating a monitor with a custom metric for response length, the team can automatically detect significant changes from the baseline and receive alerts. This is the most direct and integrated approach on Databricks for this monitoring requirement.

Answer

Mosaic AI Model Serving endpoint logs analyzed with the `mlflow.evaluate()` API on a schedule.

Answer

Databricks SQL alerts on a scheduled query that calculates the average response length from the inference table.

Answer

MLflow Model Registry webhooks to trigger a retraining pipeline when response length changes.

Question 10

A healthcare analytics team has a RAG application that must not reveal protected health information from other patients. They want to continuously monitor production traffic on their Mosaic AI Model Serving endpoint and alert when responses contain unsafe content. Which Databricks capability should they configure to evaluate each logged request and response against safety criteria and route flagged records for review?

Accepted Answer

Mosaic AI Agent Evaluation with a scheduled evaluation job over inference table data. Mosaic AI Agent Evaluation supplies judges for safety and other quality dimensions, and it can be scheduled against inference table data so every production interaction is scored. Records that fail the safety criteria can be written to a separate table for review, giving the team continuous, automated monitoring rather than manual spot checks.

Answer

Unity Catalog column masks on the inference table

Answer

MLflow autologging during model training

Answer

Delta Lake time travel on the inference table

Question 11

A team's RAG evaluation shows high context recall but low answer correctness. Retrieved documents contain the needed facts, yet the generated answers frequently contradict them. Which single metric should they examine next to pinpoint whether the generator is ignoring or misusing the retrieved context?

Accepted Answer

Groundedness, to check whether the claims in the generated answer are actually supported by the retrieved context.. With context recall high, retrieval supplied the necessary facts, so the failure lies downstream in generation. Groundedness directly measures whether the answer's claims are supported by the retrieved context, so a low groundedness score localizes the problem to the generator ignoring or misreading context. That distinguishes a generation-faithfulness issue from a retrieval-ranking or labeling issue.

Answer

Retrieval latency, to determine whether slow document fetches cause the generator to fall back on parametric memory.

Answer

Token count of the retrieved chunks, to verify whether the context window is overflowing and truncating documents.

Answer

Context precision, to check whether the most relevant chunks are ranked at the top of the retrieved list.

Question 12

A Generative AI engineer is evaluating a RAG pipeline using MLflow LLM Evaluation on Databricks. They want to assess both the retrieval quality and the generation quality. Which TWO built-in evaluation metrics should they use? (Choose two.)

Accepted Answer

`relevance`. For a RAG pipeline, retrieval quality is assessed by relevance, which measures how well retrieved documents match the query. Generation quality is assessed by groundedness, which checks if the answer is supported by the retrieved context. Both are built-in metrics in MLflow LLM Evaluation and are specifically designed for RAG evaluation on Databricks.

Answer

`perplexity`

Answer

`toxicity`

Answer

`ROUGE`

Question 13

A media company's RAG assistant answers questions about its streaming catalog. Users report that the assistant often returns answers that ignore the retrieved documents and instead rely on outdated information from the base model. The team wants an MLflow LLM Evaluation metric that measures how much of the final answer is actually derived from the retrieved context rather than the model's prior knowledge. Which metric best fits this need?

Accepted Answer

groundedness. Groundedness directly assesses whether the response is supported by the retrieved context, so a pattern of ignoring documents and using stale base-model knowledge will show up as low groundedness. That makes it the metric that quantifies the behavior the team is investigating in their catalog assistant.

Answer

relevance

Answer

context_sufficiency

Answer

answer_correctness

Question 14

A Generative AI engineer is designing an evaluation harness for a customer-support RAG agent on Databricks. They need metrics that specifically assess the RETRIEVAL stage rather than the generation stage. (Choose two.)

Accepted Answer

Context recall, measuring whether the retrieved passages collectively contain the information present in the ground-truth answer.. Context recall and context precision are computed from the retrieved passages and ground-truth relevance, so they quantify what the retriever returned and in what order. Answer relevance, groundedness, and toxicity all depend on the generated response, which means they blend generation behavior into the score. Selecting the two context-based metrics keeps the harness focused on retrieval.

Answer

Answer relevance, measuring how well the final generated response addresses the user's original question.

Answer

Toxicity, measuring whether the generated response contains harmful or offensive content.

Answer

Groundedness, measuring whether the answer's claims are supported by the retrieved context.

Question 15

A GenAI engineer notices that a RAG agent's retrieval stage is returning relevant chunks, but the generated answers frequently omit key facts present in those chunks. The team wants a single evaluation metric that isolates whether the generator is using the provided context. Which metric should they focus on?

Accepted Answer

Groundedness, which checks whether claims in the answer are supported by the retrieved context.. Groundedness is the metric that inspects whether the answer's claims are supported by the retrieved context. Since the retriever is already delivering relevant chunks, a low groundedness score points squarely at the generator ignoring or misusing that evidence, giving the team a precise diagnostic for the stage that is failing.

Answer

Answer correctness against a reference answer.

Answer

p95 latency of the serving endpoint.

Answer

Context recall of the retriever.