Databricks-GenAI-Assoc Evaluation and Monitoring — Timed 30 Questions

Question 1

An enterprise team is preparing to deploy a custom Large Language Model for a customer-facing support application on Databricks. They need to evaluate the model's responses for hallucination and groundedness using MLflow. Which specific evaluation utility should the ML Engineer utilize?

30 minute time limit — choose an answer to begin.

Accepted Answer

Invoke mlflow.evaluate() incorporating LLM-as-a-judge metrics to assess retrieved context alignment and factual consistency.. MLflow provides specialized GenAI evaluation functions, notably mlflow.evaluate(), which natively integrates LLM-judges for metrics like groundedness and hallucination using context passages. This is crucial for production deployments to ensure reliability before user exposure. Ensuring systematic metric tracking prevents deployment regressions.

Answer

Utilize traditional scikit-learn classification metrics such as F1-score and accuracy computed over raw token probabilities.

Answer

Deploy the model directly to a Databricks Model Serving endpoint and manually review a static sample of fifty chat logs.

Answer

Compute standard BLEU and ROUGE scores using standard python packages without utilizing MLflow tracking capabilities.

Question 2

An AI engineer is configuring MLflow LLM Evaluation on Databricks to assess a Retrieval-Augmented Generation (RAG) pipeline. Which TWO evaluation metrics require both the retrieved context and the ground truth answer to compute accurately? (Select TWO)

Accepted Answer

Retrieval relevance. Evaluating RAG pipelines requires distinguishing between retrieval quality and generation quality. Certain metrics rely specifically on the retrieved context chunks, whereas others evaluate answer relevance against the provided ground truth or user query. Selecting the right combination ensures comprehensive diagnostic visibility.

Answer

Relevance

Answer

Groundedness

Answer

Toxicness

Question 3

Refer to the exhibit. The engineer notices that while inference logs are being captured, the standard Databricks monitoring dashboard for the endpoint is not populating any quality metrics. What is the most likely cause?

Accepted Answer

The `monitoring_enabled` flag is set to false.. The JSON configuration indicates that while `inference_tables` are enabled, the `monitoring_enabled` flag is set to false. Databricks Model Serving requires explicit configuration to bridge inference tables with automated monitoring dashboards. Without this flag, the platform does not trigger the internal evaluation processes that calculate and display quality metrics such as drift scores or response accuracy trends within the dashboard interface.

Answer

The inference table schema is incorrectly defined.

Answer

The endpoint does not support monitoring for chat-based LLMs.

Answer

The inference table is missing write permissions.

Question 4

An organization is using 'LLM-as-a-judge' to evaluate their RAG application. What is the primary risk associated with this approach if the judge model is not carefully selected?

Accepted Answer

The evaluation process will become too fast.. Using an LLM to evaluate another LLM introduces bias and potential self-reinforcement loops, especially if the judge model is less capable or shares the same training biases as the model being evaluated. If the judge model lacks the required reasoning capability, it may systematically misgrade outputs. This creates a false sense of security where metrics look good but user experience remains poor due to flawed evaluation logic.

Answer

The judge model will always output the same score.

Answer

The cost of evaluation will be zero.

Answer

The judge model requires a separate vector database.

Question 5

Refer to the exhibit. An engineer evaluates a RAG pipeline and sees the provided results. What is the most reasonable conclusion regarding the system's performance?

Accepted Answer

The retrieval component needs investigation.. The evaluation shows high faithfulness (0.95) but low answer relevance (0.60). This indicates the model is very good at using the retrieved information correctly but is struggling to align that information with the actual user query. The retriever might be returning accurate, high-quality documents that are simply not relevant to the user's question. The engineer should investigate the retrieval logic or the embedding strategy, not the generative model itself.

Answer

The model is hallucinating frequently.

Answer

The generative model is broken and must be replaced.

Answer

The evaluation dataset is biased.

Question 6

A security auditor requires proof that the LLM application is safe for production. Which THREE of the following monitoring/evaluation tasks would satisfy this requirement? (Choose three)

Accepted Answer

Evaluate the model for compliance with system prompt instructions.. Security and safety in LLMs require a multi-faceted approach involving data grounding, prompt injection detection, and content filtering. Ensuring the model follows guardrails prevents malicious or off-topic usage, while monitoring for groundedness prevents misinformation. By combining these, you create a robust defense that demonstrates the system behaves predictably and securely under various adversarial conditions, which is exactly what a security auditor looks for in a production-ready AI implementation.

Answer

Monitor the model's GPU temperature.

Answer

Increase the number of fine-tuning epochs.

Question 7

Refer to the exhibit. An LLM-powered RAG system returns these metrics. Which action should the team prioritize to improve performance?

Accepted Answer

Implement a re-ranking mechanism to filter irrelevant documents before generation.. The low context_precision score of 0.45 indicates that the retrieval component is likely fetching irrelevant documents, leading to noise in the prompt context. By focusing on improving the retrieval mechanism, such as implementing hybrid search, re-ranking models, or optimizing chunking strategies, the system will provide higher quality data to the LLM, directly improving the precision of the generated answers without needing to retrain the model itself.

Answer

Increase the number of retrieved chunks to ensure more information is included.

Answer

Fine-tune the generative model on the domain-specific dataset.

Answer

Reduce the temperature setting of the LLM to decrease hallucinations.

Question 8

A GenAI engineer runs mlflow.evaluate() with model_type="databricks-agent" on a RAG chain registered as a Unity Catalog function. The run completes and the evaluation results table is written, but the engineer needs to compare this run against the previous production baseline and programmatically gate the next deployment on it inside a Databricks job. Which action should the engineer take?

Accepted Answer

Query the MLflow run's metrics with mlflow.search_runs() filtering on the run ID and compare the returned metric values to the baseline thresholds in the job.. MLflow LLM Evaluation records aggregate judge metrics on the evaluation run itself, so pulling them back with mlflow.search_runs() gives the job a stable numeric signal to compare against a stored baseline. Re-running evaluation, recomputing from per-row results, or mixing in serving telemetry all fail to provide that consistent, framework-defined run-level metric needed for an automated promotion gate.

Answer

Read the Delta table that stores the evaluation results and compute a mean over the per-row correctness column to derive a run-level score.

Answer

Re-run mlflow.evaluate() with the same evaluation dataset and inspect the printed summary in the notebook output to confirm the metrics match the baseline.

Answer

Enable inference tables on the serving endpoint and compare the latest logged request payloads to the baseline metric values stored in the model registry.

Question 9

A team is building an offline regression suite for a RAG agent using MLflow LLM Evaluation on Databricks. They want the evaluation dataset to support both retrieval quality and answer quality judgments and to make future runs comparable. Which TWO dataset design choices should they make? (Choose two.)

Accepted Answer

Include the expected retrieved document identifiers or relevant context for each question.. Ground-truth expected responses enable answer correctness and similarity scoring, while expected document identifiers or relevant context enable retrieval metrics such as context recall and precision. Together they let one dataset drive both stages of RAG evaluation and keep runs comparable over time as the agent and retriever evolve.

Answer

Version the evaluation dataset in Unity Catalog and pin the version used by each evaluation run.

Answer

Limit the dataset to questions the current agent already answers correctly to keep scores stable.

Answer

Store the dataset only as raw prompt strings so the judge can infer the intended answer.

Question 10

A GenAI engineer at a retail bank has deployed a RAG assistant on Mosaic AI Model Serving. Compliance requires that any answer about account balances must be grounded strictly in the retrieved policy documents. During evaluation, the engineer notices that some answers include invented interest rates not present in any retrieved chunk. Which metric should the engineer inspect FIRST to quantify this grounding failure?

Accepted Answer

Context sufficiency. The failure is that generated answers contain facts absent from the retrieved chunks. Groundedness-style metrics such as context sufficiency evaluate whether the retrieved context supports the generated claims, which is exactly the compliance requirement here. Answer relevance, latency, and token usage measure different properties and would not expose unsupported interest-rate values.

Answer

Answer relevance

Answer

Latency measured at the serving endpoint

Answer

Token usage per request

Question 11

A data engineering team is deploying a RAG application using Mosaic AI Model Serving. They need to monitor the quality of the model's responses in production. Which Databricks feature should they use to capture and analyze inference data, such as requests, responses, and latency metrics?

Accepted Answer

Mosaic AI Inference Tables. Mosaic AI Model Serving provides built-in inference tables to automatically capture request and response logs. By enabling these tables, engineers can export data to a Unity Catalog table for analysis. This is critical for monitoring performance, data drift, and model quality over time. Without this feature, teams lack the visibility required for production-grade LLM governance and continuous improvement cycles within the Databricks ecosystem.

Answer

Databricks SQL Alerts

Answer

Delta Live Tables Audit Logs

Answer

MLflow Experiment Tracking

Question 12

Which of the following metrics is most effective for evaluating a RAG-based chatbot's ability to retrieve relevant context from a vector database during production monitoring?

Accepted Answer

Context Relevance. Context relevance and faithfulness are key RAG metrics. Specifically, evaluating the retrieved context's alignment with the user query is essential for identifying retrieval failures. If the context is irrelevant, the model cannot generate a correct answer even with high-quality generative capabilities. Monitoring this metric helps distinguish between errors caused by the retriever versus errors caused by the generator model during the production lifecycle.

Answer

Model Training Loss

Answer

Inference Latency

Answer

Parameter Count

Question 13

A Generative AI engineer is configuring a Mosaic AI Model Serving endpoint for a production-grade LLM. Which TWO of the following tasks are necessary to ensure effective monitoring and evaluation of the endpoint? (Choose two)

Accepted Answer

Enable Inference Tables on the serving endpoint.. Effective monitoring requires both capturing data and evaluating that data against ground truth or automated metrics. Enabling inference tables provides the necessary raw data, while integrating with evaluation tools allows for systematic scoring of output quality. Together, these steps form the backbone of a robust monitoring strategy, ensuring that production drift is detected and quality is maintained according to business standards.

Answer

Set up an automatic retraining job on the endpoint.

Answer

Increase the GPU count to maximize inference throughput.

Answer

Delete all request logs after 24 hours to save storage.

Question 14

When evaluating a generative AI model using the Mosaic AI Model Evaluation tool, what is the primary purpose of providing a 'baseline' dataset?

Accepted Answer

To provide a reference for comparative performance analysis.. A baseline dataset serves as a ground-truth reference point. By comparing the model's current outputs against this known high-quality set, engineers can objectively measure performance degradation or improvement. This benchmarking approach is fundamental to scientific model evaluation, ensuring that updates to the model or data retrieval logic do not silently introduce regressions in the quality of the generated answers during the deployment process.

Answer

To increase the number of tokens the model can generate.

Answer

To act as a secondary training set for fine-tuning.

Answer

To optimize the vector database search latency.

Question 15

Which THREE of the following are common challenges when monitoring LLM applications in production that differ significantly from traditional ML monitoring? (Choose three)

Accepted Answer

Non-deterministic output behavior.. LLMs introduce unique challenges because their outputs are non-deterministic, high-dimensional, and often lack a single 'correct' answer. Traditional monitoring focuses on numerical drift and binary classification metrics like precision/recall. LLM evaluation requires semantic understanding, nuance assessment, and the ability to handle unstructured text, necessitating specialized tools like LLM-as-a-judge or human-in-the-loop validation to manage the subjectivity inherent in generative AI outputs.

Answer

Lack of high-throughput API endpoints.

Answer

The model's inability to connect to internet data.

Question 16

When monitoring a RAG application, you notice a high discrepancy between the retrieved context and the generated answer. Which metric would specifically help identify if the model is ignoring the provided context?

Accepted Answer

Faithfulness. Faithfulness measures whether the generated answer is derived exclusively from the retrieved context. If a model generates information not supported by the source text, it is 'hallucinating' or ignoring the context. Monitoring faithfulness is crucial for RAG systems because it directly detects when the model drifts away from the ground truth provided by the internal knowledge base, which is the primary value proposition of a RAG architecture.

Answer

Context Precision

Answer

Retrieval Recall

Answer

Semantic Similarity

Question 17

A team is using MLflow to track their model evaluation runs. They want to ensure that every evaluation run is reproducible. What is the best practice to achieve this in Databricks?

Accepted Answer

Log the model URI, dataset version, and code hash.. Reproducibility in evaluation requires logging not only the results but also the exact artifacts, environment, and code version used. MLflow's ability to log the model URI, the specific evaluation dataset version, and the code state (via git hash) ensures that an evaluation run can be perfectly recreated. This is mandatory for auditing and verifying improvements in LLM performance over time, preventing 'black box' results that cannot be validated.

Answer

Only log the final scalar metrics.

Answer

Hardcode the dataset paths in the notebook.

Answer

Use a global variable for model versioning.

Question 18

When evaluating LLM outputs, which TWO metrics are most appropriate for measuring the 'quality' of a response in a RAG system? (Choose two)

Accepted Answer

Faithfulness. Answering quality in RAG requires looking at two distinct dimensions: the correctness of the content (faithfulness) and the relevance of the answer to the user's intent. Faithfulness ensures the model remains grounded in the provided evidence, while answer relevance captures whether the model actually addressed the specific user question. These metrics provide a balanced view, ensuring the system is both accurate and useful, which is essential for end-user satisfaction.

Answer

Inference Latency

Answer

GPU Utilization

Answer

Training Iterations

Question 19

Which of the following describes the 'drift' phenomenon in the context of LLM monitoring?

Accepted Answer

A shift in user query patterns over time.. LLM drift occurs when the input data distribution or the nature of user queries changes over time, causing the model to perform worse than during validation. Because user expectations and language usage evolve, a model that performed well at launch may become less accurate as queries drift into new domains or change in tone. Monitoring this drift is essential for proactive maintenance and model updating.

Answer

The model's weights change during inference.

Answer

The model generating the same answer too often.

Answer

Increased latency in the serving endpoint.

Question 20

Why should you integrate Mosaic AI Model Evaluation with Unity Catalog?

Accepted Answer

To provide centralized governance and lineage.. Unity Catalog acts as the central governance layer for all Databricks data. By integrating evaluation with Unity Catalog, you ensure that all evaluation datasets, results, and models are discoverable, lineage-tracked, and governed. This is essential for compliance, ensuring that every model deployment is backed by a verifiable audit trail of its evaluation performance, which is a core requirement for enterprise AI deployments that must adhere to strict internal and external standards.

Answer

To increase the model's training speed.

Answer

To automatically delete old evaluation data.

Answer

To bypass the need for prompt engineering.