Databricks-GenAI-Assoc Evaluation and Monitoring — All Questions With Answers

Question 1mediummultiple choice

Read the full Evaluation and Monitoring explanation →

An enterprise team is preparing to deploy a custom Large Language Model for a customer-facing support application on Databricks. They need to evaluate the model's responses for hallucination and groundedness using MLflow. Which specific evaluation utility should the ML Engineer utilize?

Question 2hardmulti select

Read the full Evaluation and Monitoring explanation →

An AI engineer is configuring MLflow LLM Evaluation on Databricks to assess a Retrieval-Augmented Generation (RAG) pipeline. Which TWO evaluation metrics require both the retrieved context and the ground truth answer to compute accurately? (Select TWO)

Question 3mediummultiple choice

Read the full Evaluation and Monitoring explanation →

Refer to the exhibit. The engineer notices that while inference logs are being captured, the standard Databricks monitoring dashboard for the endpoint is not populating any quality metrics. What is the most likely cause?

Exhibit

{
  "endpoint_name": "llm-chat-bot",
  "inference_tables": {
    "enabled": true,
    "catalog": "main",
    "schema": "monitoring",
    "table_name": "inference_logs"
  },
  "monitoring_enabled": false
}

Question 4hardmultiple choice

Read the full Evaluation and Monitoring explanation →

An organization is using 'LLM-as-a-judge' to evaluate their RAG application. What is the primary risk associated with this approach if the judge model is not carefully selected?

Question 5hardmultiple choice

Read the full Evaluation and Monitoring explanation →

Refer to the exhibit. An engineer evaluates a RAG pipeline and sees the provided results. What is the most reasonable conclusion regarding the system's performance?

Exhibit

{
  "evaluation_run": "eval-789",
  "model": "gpt-4-turbo",
  "dataset": "validation_set_v2",
  "results": {
    "faithfulness": 0.95,
    "answer_relevance": 0.60
  }
}

Question 6hardmulti select

Read the full Evaluation and Monitoring explanation →

A security auditor requires proof that the LLM application is safe for production. Which THREE of the following monitoring/evaluation tasks would satisfy this requirement? (Choose three)

Question 7hardmultiple choice

Read the full Evaluation and Monitoring explanation →

Refer to the exhibit. An LLM-powered RAG system returns these metrics. Which action should the team prioritize to improve performance?

Exhibit

{
  "evaluation_metrics": {
    "ground_truth_relevance": 0.82,
    "context_precision": 0.45,
    "faithfulness": 0.91
  }
}

Question 8hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A GenAI engineer runs mlflow.evaluate() with model_type="databricks-agent" on a RAG chain registered as a Unity Catalog function. The run completes and the evaluation results table is written, but the engineer needs to compare this run against the previous production baseline and programmatically gate the next deployment on it inside a Databricks job. Which action should the engineer take?

Question 9mediummulti select

Read the full Evaluation and Monitoring explanation →

A team is building an offline regression suite for a RAG agent using MLflow LLM Evaluation on Databricks. They want the evaluation dataset to support both retrieval quality and answer quality judgments and to make future runs comparable. Which TWO dataset design choices should they make? (Choose two.)

Question 10mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A GenAI engineer at a retail bank has deployed a RAG assistant on Mosaic AI Model Serving. Compliance requires that any answer about account balances must be grounded strictly in the retrieved policy documents. During evaluation, the engineer notices that some answers include invented interest rates not present in any retrieved chunk. Which metric should the engineer inspect FIRST to quantify this grounding failure?

Question 11mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A data engineering team is deploying a RAG application using Mosaic AI Model Serving. They need to monitor the quality of the model's responses in production. Which Databricks feature should they use to capture and analyze inference data, such as requests, responses, and latency metrics?

Question 12mediummultiple choice

Read the full Evaluation and Monitoring explanation →

Which of the following metrics is most effective for evaluating a RAG-based chatbot's ability to retrieve relevant context from a vector database during production monitoring?

Question 13hardmulti select

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer is configuring a Mosaic AI Model Serving endpoint for a production-grade LLM. Which TWO of the following tasks are necessary to ensure effective monitoring and evaluation of the endpoint? (Choose two)

Question 14easymultiple choice

Read the full Evaluation and Monitoring explanation →

When evaluating a generative AI model using the Mosaic AI Model Evaluation tool, what is the primary purpose of providing a 'baseline' dataset?

Question 15mediummulti select

Read the full Evaluation and Monitoring explanation →

Which THREE of the following are common challenges when monitoring LLM applications in production that differ significantly from traditional ML monitoring? (Choose three)

Question 16mediummultiple choice

Read the full Evaluation and Monitoring explanation →

When monitoring a RAG application, you notice a high discrepancy between the retrieved context and the generated answer. Which metric would specifically help identify if the model is ignoring the provided context?

Question 17hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A team is using MLflow to track their model evaluation runs. They want to ensure that every evaluation run is reproducible. What is the best practice to achieve this in Databricks?

Question 18mediummulti select

Read the full Evaluation and Monitoring explanation →

When evaluating LLM outputs, which TWO metrics are most appropriate for measuring the 'quality' of a response in a RAG system? (Choose two)

Question 19mediummultiple choice

Read the full Evaluation and Monitoring explanation →

Which of the following describes the 'drift' phenomenon in the context of LLM monitoring?

Question 20easymultiple choice

Read the full Evaluation and Monitoring explanation →

Why should you integrate Mosaic AI Model Evaluation with Unity Catalog?

Question 21mediummultiple choice

Read the full Evaluation and Monitoring explanation →

An enterprise LLM application on Databricks is experiencing high latency and inconsistent responses. Which approach best enables observability to identify the root cause of these performance bottlenecks within the LLM pipeline?

Question 22mediummulti select

Read the full Evaluation and Monitoring explanation →

Which TWO evaluation approaches are most effective for measuring the quality of a RAG pipeline's retrieval stage?

Question 23easymultiple choice

Read the full Evaluation and Monitoring explanation →

What is the primary benefit of using MLflow Model Evaluation for generative AI applications compared to manual evaluation methods?

Question 24mediummultiple choice

Read the full Evaluation and Monitoring explanation →

When monitoring a production LLM, you detect a drift where the model's responses are becoming increasingly verbose and less helpful compared to the baseline. Which strategy is most effective for detecting this quality decay?

Question 25hardmulti select

Read the full Evaluation and Monitoring explanation →

Which THREE components are critical to include in a comprehensive evaluation strategy for a RAG-based Generative AI application?

Question 26mediummultiple choice

Read the full Evaluation and Monitoring explanation →

Your organization requires an auditable record of all LLM evaluation results for compliance. Which Databricks feature provides the best centralized storage for these evaluation runs?

Question 27mediummultiple choice

Read the full Evaluation and Monitoring explanation →

Refer to the exhibit. An engineer observes an unexpected drop in "relevance" for the latest deployment. What is the most likely cause related to the evaluation process itself?

Exhibit

{
  "model_eval": {
    "name": "prod_chatbot_v2",
    "eval_data": "eval_set_v4",
    "results": {
      "faithfulness": 0.95,
      "relevance": 0.88
    }
  }
}

Question 28mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A Databricks Generative AI engineer has deployed a RAG application and is now setting up production monitoring. They want to automatically detect when the distribution of incoming user questions diverges from the distribution seen during development, so they can trigger retraining or prompt adjustments. Which Databricks capability should they configure?

Question 29hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A team is using MLflow LLM Evaluation with the built-in answer_correctness metric to compare two prompt templates for a question-answering application. They notice that answer_correctness scores are nearly identical, but manual review shows one template produces answers that are factually correct yet omit key supporting details. Which additional built-in metric should they add to their evaluation to surface this difference?

Question 30easymultiple choice

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer is deploying a new version of a RAG chain to a Mosaic AI Model Serving endpoint. Before promoting it to production, they want to run an evaluation that checks whether the generated answers are faithful to the retrieved documents. Which Mosaic AI Agent Evaluation metric should they examine?

Question 31mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A GenAI engineering team has deployed a customer-support RAG chain on Databricks and registered it in Unity Catalog. They now want MLflow 3 to automatically score every production request for groundedness and relevance without writing custom scoring code, and to persist those assessments against the logged traces. Which approach should they use?

Question 32mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A team runs an LLM-as-a-judge evaluation on Databricks using the built-in `mlflow.evaluate()` with `model_type="databricks-agent"`. They notice that the judge model, a serving endpoint, is producing scores that are consistently inflated compared to human review. Which configuration change should the team make first to improve the reliability of the evaluation?

Question 33hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A team runs a RAG application on Mosaic AI Model Serving and logs all requests to an inference table. Reviewers report that answers are sometimes fluent but contradict the retrieved documents. The team wants a recurring, automated check that quantifies this contradiction on production traffic and alerts when it exceeds a threshold. Which approach best fits?

Question 34easymultiple choice

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer at a retail company has deployed a RAG chatbot on a Databricks Mosaic AI Model Serving endpoint. The team wants to capture per-request evaluation data, including the user's question, the retrieved context chunks, and the model's response, so they can analyze response quality over time. Which Databricks feature should they use to collect this data for downstream evaluation?

Question 35mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A team runs a RAG chatbot whose MLflow evaluation with the built-in groundedness judge previously scored well. After they swap the retriever for a new embedding model, groundedness scores drop sharply even though the generator model and prompt are unchanged. They confirm the judge model itself is unchanged. Which action should they take FIRST to diagnose the regression?

Question 36hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer is using MLflow Tracing to monitor a RAG application deployed on Mosaic AI Model Serving. They want to capture the retrieved documents, the final prompt, and the model's response for each request to debug a quality issue. Which approach should they use to ensure all three are logged in a single trace?

Question 37mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A financial services team runs a RAG assistant on Databricks that answers questions about internal policy documents. During a review, the team finds that for many questions the answer is factually correct but cites a document that does not actually contain the supporting statement. They want an MLflow LLM Evaluation metric that specifically detects when the response is not supported by the retrieved context. Which metric should they add to their evaluation run?

Question 38hardmultiple choice

Read the full Evaluation and Monitoring explanation →

An engineer registers a GenAI agent to Unity Catalog and enables inference tables on its Mosaic AI Model Serving endpoint. They want to automatically detect when response quality degrades in production without waiting for human review. Which capability should they configure to achieve continuous automated quality monitoring?

Question 39mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A team has deployed a RAG chatbot on Databricks and enabled inference table logging. They want to set up automated monitoring to detect when the average response length increases significantly compared to the baseline. Which Databricks feature should they use to create this monitor?

Question 40hardmultiple choice

Review the full routing breakdown →

A healthcare analytics team has a RAG application that must not reveal protected health information from other patients. They want to continuously monitor production traffic on their Mosaic AI Model Serving endpoint and alert when responses contain unsafe content. Which Databricks capability should they configure to evaluate each logged request and response against safety criteria and route flagged records for review?

Question 41mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A team's RAG evaluation shows high context recall but low answer correctness. Retrieved documents contain the needed facts, yet the generated answers frequently contradict them. Which single metric should they examine next to pinpoint whether the generator is ignoring or misusing the retrieved context?

Question 42hardmulti select

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer is evaluating a RAG pipeline using MLflow LLM Evaluation on Databricks. They want to assess both the retrieval quality and the generation quality. Which TWO built-in evaluation metrics should they use? (Choose two.)

Question 43mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A media company's RAG assistant answers questions about its streaming catalog. Users report that the assistant often returns answers that ignore the retrieved documents and instead rely on outdated information from the base model. The team wants an MLflow LLM Evaluation metric that measures how much of the final answer is actually derived from the retrieved context rather than the model's prior knowledge. Which metric best fits this need?

Question 44hardmulti select

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer is designing an evaluation harness for a customer-support RAG agent on Databricks. They need metrics that specifically assess the RETRIEVAL stage rather than the generation stage. (Choose two.)

Question 45easymultiple choice

Read the full Evaluation and Monitoring explanation →

A GenAI engineer notices that a RAG agent's retrieval stage is returning relevant chunks, but the generated answers frequently omit key facts present in those chunks. The team wants a single evaluation metric that isolates whether the generator is using the provided context. Which metric should they focus on?

Question 46hardmulti select

Read the full Evaluation and Monitoring explanation →

A Generative AI engineer is setting up MLflow LLM Evaluation for a RAG pipeline on Databricks that answers questions about product manuals. The team wants to evaluate both the retrieval stage and the generation stage in a single evaluation run. Which TWO metric groups should they include in the evaluation configuration? (Choose two.)

Question 47easymultiple choice

Read the full Evaluation and Monitoring explanation →

A team has deployed a RAG application on Databricks and wants to monitor the quality of responses in production. They have enabled inference table logging. Which built-in Databricks capability allows them to periodically evaluate the logged requests and responses for quality metrics like groundedness?

Question 48easymultiple choice

Read the full Evaluation and Monitoring explanation →

A team wants every MLflow evaluation run of their RAG chain to be reproducible months later, including the exact prompt template, model parameters, and retrieved context used. Which practice best ensures this reproducibility?

Question 49hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A team runs an offline evaluation of a RAG chatbot using MLflow LLM Evaluation with an LLM judge. The same evaluation dataset produces noticeably different scores when re-run on different days, even though the application code and retrieved documents are unchanged. Which action best addresses this score instability?

Question 50hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A team uses an LLM judge to score their RAG agent on Databricks and sees scores that fluctuate by several points between identical evaluation runs. They need more stable, reproducible quality signals for release gating. Which action best addresses the root cause?

Question 51mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A GenAI team at a retail bank runs a RAG assistant on a Databricks Mosaic AI Model Serving endpoint. During a pilot, they captured end-user thumbs-up/down feedback in a Delta table but never joined it to the trace payloads. Six weeks later, hallucination complaints spike, yet the aggregate thumbs-down rate is unchanged. Which approach best resolves this discrepancy using Databricks-native tooling?

Question 52mediummulti select

Read the full Evaluation and Monitoring explanation →

A GenAI engineer is instrumenting a production RAG application on Databricks to detect quality degradation before users complain. The team wants signals that reveal problems in the retrieval stage specifically, rather than in the generation stage. Which TWO signals should they track? (Choose two.)

Question 53hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A healthcare analytics team deployed a RAG assistant on Databricks whose answers cite clinical policy documents. Compliance requires that every production answer be attributable to a specific retrieved chunk. During evaluation, the team notices the groundedness judge scores are high, but manual review finds answers that blend two policies into a statement neither document supports. Which change to their Mosaic AI Agent Evaluation configuration best detects this failure mode?

Question 54easymultiple choice

Read the full Evaluation and Monitoring explanation →

A support team operates a Databricks-hosted RAG assistant and wants end-user feedback to feed their monitoring dashboards. They plan to add a thumbs-up and thumbs-down control to the chat UI and log each vote alongside the request ID. What is the primary value of collecting this feedback for the evaluation and monitoring workflow?

Question 55mediummultiple choice

Read the full Evaluation and Monitoring explanation →

A generative AI team runs a RAG chatbot whose Mosaic AI Model Serving endpoint is monitored in Unity Catalog inference tables. Over two weeks, the percentage of user questions that receive a refusal answer ('I don't have enough information') climbs from 4% to 31%, while retrieval latency and token counts stay flat. The team wants the earliest actionable signal that the retrieval corpus has gone stale rather than the prompt or model. Which monitoring signal should they inspect first?

Question 56mediummulti select

Read the full Evaluation and Monitoring explanation →

A fintech company runs a customer-support RAG assistant on Databricks. Before promoting a new prompt template from staging to production, the ML team must demonstrate that the change does not regress answer quality. Which TWO evaluation practices should they apply in Mosaic AI Agent Evaluation to make this promotion decision defensible? (Choose two.)

Question 57hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A GenAI engineer monitors a customer-facing RAG assistant hosted on Databricks. After a routine re-indexing job, groundedness scores from the MLflow LLM Evaluation job drop sharply while answer relevance stays flat. The application prompt and the LLM serving endpoint were untouched. Which conclusion is best supported by these signals?

Question 58hardmultiple choice

Read the full Evaluation and Monitoring explanation →

An engineer uses MLflow LLM Evaluation with mlflow.evaluate() to score a RAG application. The judge model is configured with a temperature of 0.9 and no explicit metric thresholds are set. Reruns of the identical evaluation dataset produce relevance scores that swing by up to 20 percentage points, and the team cannot tell whether a prompt change helped. Which change most directly improves the reliability of the evaluation comparison?

Question 59easymultiple choice

Read the full Evaluation and Monitoring explanation →

A logistics company monitors a Databricks-hosted RAG assistant that answers questions about shipping regulations. Over one week, the retrieval index was rebuilt after a document refresh, and the team observes that the rate of answers flagged as unsupported by retrieved context rose sharply while retrieval relevance scores stayed flat. Which metric should they inspect first to determine whether the regression originates in the retrieval stage or the generation stage?

Question 60easymultiple choice

Read the full Evaluation and Monitoring explanation →

A team deploys a customer-support assistant on a Mosaic AI Model Serving endpoint and enables inference table logging to Unity Catalog. Compliance requires that every production response be traceable back to the exact request, the retrieved context, and the model version that produced it, and that reviewers can query this history with SQL months later. Which capability satisfies this requirement?

Question 61mediummulti select

Read the full Evaluation and Monitoring explanation →

A RAG assistant built with Mosaic AI Agent Framework is in production. Product managers complain that answers are sometimes plausible but unsupported by the retrieved documents, and separately that the assistant sometimes ignores documents that clearly contain the answer. The team wants automated, scheduled checks in MLflow LLM Evaluation that separate these two failure modes so each can be triaged independently. Which TWO evaluation metrics should they add to the evaluation suite? (Choose two.)

Question 62hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A media company uses an LLM-as-a-judge evaluation pipeline on Databricks to score a summarization assistant nightly. The judge is the same model family as the assistant and was given a rubric that rewards stylistic fluency. Over two months, nightly scores climb steadily while editor spot-checks find summaries increasingly omit key facts. Which corrective action best restores the evaluation's ability to detect this regression?

Question 63hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A team uses MLflow LLM Evaluation with an LLM judge to score a summarization agent. On a 400-row golden dataset the judge marks 96% of summaries as relevant. Spot-checking reveals the judge approves nearly every summary whenever the summary is fluent, even when key facts are missing. The team wants a defensible quality signal before approving a release. Which action best addresses this judge weakness?

Question 64hardmultiple choice

Read the full Evaluation and Monitoring explanation →

A retail company runs a customer-facing RAG assistant on a Mosaic AI Model Serving endpoint. The team wants every production request, response, and retrieved context to be captured automatically into a Unity Catalog Delta table so they can monitor quality and latency trends over time. Which action should they take?

{ "endpoint_name": "llm-chat-bot", "inference_tables": { "enabled": true, "catalog": "main", "schema": "monitoring", "table_name": "inference_logs" }, "monitoring_enabled": false }