Databricks-GenAI-Assoc Evaluation and Monitoring Practice Test 4 — 15 Questions

Question 1

A Generative AI engineer is setting up MLflow LLM Evaluation for a RAG pipeline on Databricks that answers questions about product manuals. The team wants to evaluate both the retrieval stage and the generation stage in a single evaluation run. Which TWO metric groups should they include in the evaluation configuration? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Retrieval metrics such as retrieval_relevance and retrieval_groundedness. A complete RAG evaluation run needs metrics for both stages. Retrieval metrics score the chunks returned by the retriever, while generation metrics such as groundedness and relevance score the final answer against the question and context. Together they let the team pinpoint whether a quality issue originates in retrieval or in generation.

Answer

System metrics such as CPU utilization and memory consumption of the serving endpoint

Answer

Training metrics such as loss and learning rate from the fine-tuning job

Answer

Data freshness metrics such as table update timestamps in Unity Catalog

Question 2

A team has deployed a RAG application on Databricks and wants to monitor the quality of responses in production. They have enabled inference table logging. Which built-in Databricks capability allows them to periodically evaluate the logged requests and responses for quality metrics like groundedness?

Accepted Answer

Mosaic AI Agent Evaluation, which can be run on the inference table to compute quality metrics.. Mosaic AI Agent Evaluation is the built-in Databricks capability for evaluating RAG and agent applications. It can process inference tables and compute quality metrics such as groundedness, relevance, and answer correctness. This enables periodic, automated monitoring of production quality without custom code. It is the correct tool for the described scenario.

Answer

Databricks SQL dashboards that display the count of requests per hour.

Answer

Delta Live Tables expectations that validate response length.

Answer

MLflow Tracking with custom metrics logged during model serving.

Question 3

A team wants every MLflow evaluation run of their RAG chain to be reproducible months later, including the exact prompt template, model parameters, and retrieved context used. Which practice best ensures this reproducibility?

Accepted Answer

Log the evaluation as an MLflow run with the model parameters, prompt template, and input dataset version recorded as run metadata and artifacts.. Reproducibility depends on capturing inputs and configuration, not just scores. Logging the run with parameters, the prompt template, and a versioned dataset as artifacts and metadata ties every result to the exact conditions that produced it. That lets anyone rerun or audit the evaluation later, even after library upgrades or dataset changes.

Answer

Export the evaluation results to a CSV file stored on a personal laptop so the engineer can compare future runs manually.

Answer

Rely on the built-in default judge prompts so that no prompt configuration needs to be tracked between runs.

Answer

Store only the final aggregate metric values in a shared spreadsheet, since the underlying outputs can always be regenerated on demand.

Question 4

A team runs an offline evaluation of a RAG chatbot using MLflow LLM Evaluation with an LLM judge. The same evaluation dataset produces noticeably different scores when re-run on different days, even though the application code and retrieved documents are unchanged. Which action best addresses this score instability?

Accepted Answer

Pin the judge model version and set temperature to zero for evaluation runs. Identical inputs producing different scores across runs indicate the judge itself is nondeterministic, typically through model version drift or sampling temperature. Pinning the judge model version and forcing temperature to zero makes scoring reproducible, so observed differences can be attributed to the application. The other options either change the system under test, destroy evaluation fidelity, or hide variance rather than eliminate it.

Answer

Increase the number of retrieved chunks per query so the judge has more context

Answer

Switch all metrics from LLM-judged to exact-match string comparison

Answer

Reduce the evaluation dataset to a single representative example

Question 5

A team uses an LLM judge to score their RAG agent on Databricks and sees scores that fluctuate by several points between identical evaluation runs. They need more stable, reproducible quality signals for release gating. Which action best addresses the root cause?

Accepted Answer

Pin the judge model version and set its temperature to zero, then validate the judge against a human-labeled sample.. Fluctuating judge scores usually stem from nondeterministic sampling and drifting judge model versions. Pinning the judge version and setting temperature to zero makes each scoring pass reproducible, and validating the judge against human labels confirms that the stabilized scores are also accurate enough to gate releases reliably.

Answer

Increase the number of evaluation examples until the average stabilizes across runs.

Answer

Average the scores from three different judge models and use the mean for gating.

Answer

Switch from an LLM judge to BLEU or ROUGE string-overlap metrics.

Question 6

A GenAI team at a retail bank runs a RAG assistant on a Databricks Mosaic AI Model Serving endpoint. During a pilot, they captured end-user thumbs-up/down feedback in a Delta table but never joined it to the trace payloads. Six weeks later, hallucination complaints spike, yet the aggregate thumbs-down rate is unchanged. Which approach best resolves this discrepancy using Databricks-native tooling?

Accepted Answer

Join the Delta feedback table to MLflow traces on request_id and segment quality metrics by user cohort, prompt template version, and retrieved document source.. Aggregate feedback rates are diluted by cohort mix, so an unchanged overall thumbs-down percentage can coexist with a severe regression inside one prompt template version, document source, or user cohort. Correlating the stored human feedback with MLflow traces on request_id and then slicing by those dimensions surfaces where the hallucination spike actually lives, which is the prerequisite for any targeted fix.

Answer

Increase the endpoint's provisioned concurrency and enable autoscaling so responses are returned faster during peak hours.

Answer

Switch the judge from a smaller open model to a larger frontier model and re-run the nightly evaluation job over the last 30 days.

Answer

Retrain the embedding model on the latest product documentation and redeploy the index to the Vector Search endpoint.

Question 7

A GenAI engineer is instrumenting a production RAG application on Databricks to detect quality degradation before users complain. The team wants signals that reveal problems in the retrieval stage specifically, rather than in the generation stage. Which TWO signals should they track? (Choose two.)

Accepted Answer

Recall of the retrieval step measured against a curated question-to-document mapping. Retrieval-stage quality is measured by whether the right documents are found and how many of the returned ones are relevant. Precision at k and recall against a curated question-to-document mapping capture those two dimensions directly, so regressions localize to indexing, chunking, or embeddings. Token count, time to first token, and toxicity all describe the generator or the serving layer, not retrieval.

Answer

Toxicity score of the final generated answer

Answer

Average completion token count per response

Answer

Time to first token on the serving endpoint

Question 8

A healthcare analytics team deployed a RAG assistant on Databricks whose answers cite clinical policy documents. Compliance requires that every production answer be attributable to a specific retrieved chunk. During evaluation, the team notices the groundedness judge scores are high, but manual review finds answers that blend two policies into a statement neither document supports. Which change to their Mosaic AI Agent Evaluation configuration best detects this failure mode?

Accepted Answer

Enable the chunk-level citation assessment so each claim in the response is checked against the specific retrieved chunk it references, rather than scoring the response against the concatenated context.. The compliance requirement is per-answer attribution to a specific chunk, so the evaluation must verify each claim against the chunk it cites. Scoring the whole response against concatenated context allows a blended claim to pass whenever its components appear somewhere in the pool. Chunk-level citation assessment closes that gap by testing attribution directly, which is what distinguishes this failure from ordinary ungrounded hallucination.

Answer

Increase the number of retrieved chunks per query from five to twenty so the model has more supporting evidence available.

Answer

Raise the temperature setting on the served model to zero and re-run the evaluation suite to obtain more deterministic outputs.

Answer

Add a relevance judge that scores the retrieved chunks against the user question and filter out any chunk scoring below a fixed threshold.

Question 9

A support team operates a Databricks-hosted RAG assistant and wants end-user feedback to feed their monitoring dashboards. They plan to add a thumbs-up and thumbs-down control to the chat UI and log each vote alongside the request ID. What is the primary value of collecting this feedback for the evaluation and monitoring workflow?

Accepted Answer

It provides real-world signal that can be joined to traces for triage and dataset curation. End-user thumbs votes are weak but real labels tied to specific production requests. Joined with traces and request IDs, they drive triage of failing interactions and curation of new offline evaluation examples, connecting production monitoring to the evaluation dataset. They do not replace datasets, train models automatically, or improve metrics on their own.

Answer

It guarantees that groundedness scores will improve over time

Answer

It replaces the need for any offline evaluation dataset

Answer

It automatically fine-tunes the underlying foundation model

Question 10

A generative AI team runs a RAG chatbot whose Mosaic AI Model Serving endpoint is monitored in Unity Catalog inference tables. Over two weeks, the percentage of user questions that receive a refusal answer ('I don't have enough information') climbs from 4% to 31%, while retrieval latency and token counts stay flat. The team wants the earliest actionable signal that the retrieval corpus has gone stale rather than the prompt or model. Which monitoring signal should they inspect first?

Accepted Answer

The distribution of retrieval scores for the top-k chunks returned by the Vector Search index, tracked over time in the inference table.. Refusals rising while latency and token counts hold steady isolates the problem to retrieval quality rather than the model or the serving tier. Retrieval score distributions logged to the inference table show whether top-k chunks are becoming weaker matches, which is the earliest signal that the indexed corpus no longer covers current user questions. Checking capacity, token cost, or generation time cannot distinguish stale content from a healthy pipeline.

Answer

The average generation time per completion recorded by the serving endpoint's request logs.

Answer

The endpoint's provisioned concurrency utilization and queue depth metrics from the serving endpoint logs.

Answer

The count of prompt tokens consumed per request, compared week over week in the billing usage table.

Question 11

A fintech company runs a customer-support RAG assistant on Databricks. Before promoting a new prompt template from staging to production, the ML team must demonstrate that the change does not regress answer quality. Which TWO evaluation practices should they apply in Mosaic AI Agent Evaluation to make this promotion decision defensible? (Choose two.)

Accepted Answer

Pin the evaluation dataset, judge model version, and retrieval configuration so both templates are scored under identical conditions and results are reproducible.. A defensible promotion gate requires a controlled comparison where the prompt template is the only variable, which means running both templates over the same dataset with the judge model and retrieval configuration pinned, and inspecting per-question deltas instead of relying on averages that can mask localized regressions. Together these practices attribute any score movement to the change under test and make the decision reproducible.

Answer

Deploy the candidate template to a small percentage of production traffic and rely on user thumbs feedback alone as the promotion signal.

Answer

Increase the judge model size until its scores on the candidate template exceed the production template's scores.

Answer

Evaluate only the questions where the candidate template produced different answers, skipping the rest to save judge tokens.

Question 12

A GenAI engineer monitors a customer-facing RAG assistant hosted on Databricks. After a routine re-indexing job, groundedness scores from the MLflow LLM Evaluation job drop sharply while answer relevance stays flat. The application prompt and the LLM serving endpoint were untouched. Which conclusion is best supported by these signals?

Accepted Answer

The re-indexing job degraded retrieval, so the generator receives weaker supporting context. The drop begins exactly when the index was rebuilt, and relevance remains flat, which isolates the change to the evidence supplied to the generator rather than to the prompt, the LLM, or the judge. Degraded retrieval after re-indexing, from altered chunking, embedding mismatches, or an incomplete build, is the best-supported conclusion and the right place to investigate first.

Answer

The LLM judge model was silently upgraded and is scoring more harshly

Answer

Users are asking harder questions than before, lowering groundedness

Answer

The generation prompt has drifted and needs to be rewritten

Question 13

An engineer uses MLflow LLM Evaluation with mlflow.evaluate() to score a RAG application. The judge model is configured with a temperature of 0.9 and no explicit metric thresholds are set. Reruns of the identical evaluation dataset produce relevance scores that swing by up to 20 percentage points, and the team cannot tell whether a prompt change helped. Which change most directly improves the reliability of the evaluation comparison?

Accepted Answer

Set the judge model's temperature to 0 and define fixed pass/fail thresholds for each metric before comparing runs.. Score swings across identical runs are caused by nondeterministic judge sampling, so the fix must remove that randomness and make the decision rule stable. Setting the judge temperature to zero makes each judgment reproducible, and predefined thresholds turn noisy continuous scores into consistent pass/fail outcomes. Enlarging the judge, duplicating rows, or hiding per-row results leaves the stochasticity in place and keeps run-to-run comparisons unreliable.

Answer

Increase the evaluation dataset size by duplicating each question three times so variance averages out across more judge calls.

Answer

Switch the evaluation to a larger judge model with more parameters while keeping the existing sampling settings.

Answer

Log only the aggregate mean relevance score per run and discard the per-row judgments from the MLflow run artifacts.

Question 14

A logistics company monitors a Databricks-hosted RAG assistant that answers questions about shipping regulations. Over one week, the retrieval index was rebuilt after a document refresh, and the team observes that the rate of answers flagged as unsupported by retrieved context rose sharply while retrieval relevance scores stayed flat. Which metric should they inspect first to determine whether the regression originates in the retrieval stage or the generation stage?

Accepted Answer

Context recall, measured against a labeled set of questions with known ground-truth source documents.. Context recall against a labeled ground-truth set directly tests whether the retriever brought back the documents that contain the answer. Because relevance measures topical similarity and can stay high even when authoritative chunks are missed, only recall can reveal that the rebuilt index now returns plausible but non-supporting passages, which would explain unsupported answers without any change in generation behavior.

Answer

The number of thumbs-up reactions collected from end users during the same week.

Answer

Endpoint request latency, comparing the p95 before and after the index rebuild.

Answer

Token usage per response, broken down by prompt and completion tokens on the serving endpoint.

Question 15

A team deploys a customer-support assistant on a Mosaic AI Model Serving endpoint and enables inference table logging to Unity Catalog. Compliance requires that every production response be traceable back to the exact request, the retrieved context, and the model version that produced it, and that reviewers can query this history with SQL months later. Which capability satisfies this requirement?

Accepted Answer

The endpoint's request and response payloads written to a Unity Catalog inference table, queried with SQL and joined to model version metadata.. Compliance-grade traceability demands durable, queryable records of each request, the context supplied to the model, the response, and the serving model version. Unity Catalog inference tables store exactly that as governed Delta tables, so SQL queries and joins to model version metadata answer audit questions months after the traffic occurred. Client logs, offline experiment runs, and aggregate dashboards all lack either the payload detail or the retention and governance needed.

Answer

Real-time dashboards built on the endpoint's built-in metrics such as request count and error rate.

Answer

MLflow experiment runs recorded during offline evaluation of the assistant before release.

Answer

Client-side application logs streamed to a log analytics workspace and retained for thirty days.