Databricks-GenAI-Assoc Practice Test 19 — 15 Questions

Question 1

A GenAI engineer is using MLflow to track experiments for a fine-tuned language model. The engineer wants to ensure that model artifacts and parameters are governed by Unity Catalog, so that access can be controlled and audited. Which Unity Catalog object should the engineer use to register the model?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

A Unity Catalog registered model. Unity Catalog registered models are the correct object for governing machine learning models. They provide a three-level namespace and support fine-grained access control, versioning, and auditing. By registering the fine-tuned model as a Unity Catalog model, the engineer can manage access and track usage, satisfying the governance requirement.

Answer

A Unity Catalog schema

Answer

A Unity Catalog volume

Answer

A Unity Catalog table

Question 2

A GenAI platform team is preparing a Unity Catalog schema to host a retrieval-augmented generation pipeline. They will store prompt templates and evaluation datasets as Delta tables, and they need to expose the pipeline to a group of application developers. The security team asks the platform team to describe how Unity Catalog privileges must be granted for the developers to query the tables. Which TWO statements correctly describe the required privilege model? (Choose two.)

Accepted Answer

Developers need USE CATALOG on the catalog and USE SCHEMA on the schema before any table privileges take effect. Reading a table in Unity Catalog requires traversing the namespace and holding a data privilege. USE CATALOG on the catalog and USE SCHEMA on the schema make the table addressable, and SELECT on the table, or on the schema for broad coverage, authorizes the read. Together these two statements describe the minimum privilege set that lets the developers query the RAG pipeline's tables without over-granting administrative rights.

Answer

Developers need CREATE TABLE on the catalog to read prompt templates stored as Delta tables

Answer

Developers must be granted ALL PRIVILEGES on the catalog so that future objects are automatically accessible

Answer

Developers must own the schema so that they can create their own tables without further grants

Question 3

A team has deployed a RAG application on Databricks and wants to monitor the quality of responses in production. They have enabled inference table logging. Which built-in Databricks capability allows them to periodically evaluate the logged requests and responses for quality metrics like groundedness?

Accepted Answer

Mosaic AI Agent Evaluation, which can be run on the inference table to compute quality metrics.. Mosaic AI Agent Evaluation is the built-in Databricks capability for evaluating RAG and agent applications. It can process inference tables and compute quality metrics such as groundedness, relevance, and answer correctness. This enables periodic, automated monitoring of production quality without custom code. It is the correct tool for the described scenario.

Answer

Databricks SQL dashboards that display the count of requests per hour.

Answer

Delta Live Tables expectations that validate response length.

Answer

MLflow Tracking with custom metrics logged during model serving.

Question 4

A financial services team runs a RAG assistant on Databricks that answers questions about internal policy documents. During a review, the team finds that for many questions the answer is factually correct but cites a document that does not actually contain the supporting statement. They want an MLflow LLM Evaluation metric that specifically detects when the response is not supported by the retrieved context. Which metric should they add to their evaluation run?

Accepted Answer

groundedness. Groundedness evaluates whether the response's claims can be traced back to the retrieved context, which is precisely the gap identified in this review. Adding it to the MLflow LLM Evaluation run gives the team a metric that flags answers which sound correct but are not actually supported by the cited policy documents.

Answer

answer_similarity

Answer

retrieval_precision

Answer

toxicity

Question 5

A team has deployed a RAG chatbot on Databricks and enabled inference table logging. They want to set up automated monitoring to detect when the average response length increases significantly compared to the baseline. Which Databricks feature should they use to create this monitor?

Accepted Answer

Lakehouse Monitoring for the inference table, with a custom metric for response length.. Lakehouse Monitoring is designed to monitor Delta tables, including inference tables, for data quality and drift. By creating a monitor with a custom metric for response length, the team can automatically detect significant changes from the baseline and receive alerts. This is the most direct and integrated approach on Databricks for this monitoring requirement.

Answer

Mosaic AI Model Serving endpoint logs analyzed with the `mlflow.evaluate()` API on a schedule.

Answer

Databricks SQL alerts on a scheduled query that calculates the average response length from the inference table.

Answer

MLflow Model Registry webhooks to trigger a retraining pipeline when response length changes.

Question 6

A GenAI team is using MLflow to track experiments for a large language model. They want to ensure that only team members can view the experiment results and that the experiment artifacts are stored in a governed location. Which Unity Catalog integration should they use to manage MLflow experiments?

Accepted Answer

Register the MLflow experiment in Unity Catalog and grant appropriate privileges on the experiment object. Registering an MLflow experiment in Unity Catalog makes it a securable object, allowing you to grant privileges to specific groups. Artifacts are stored in a governed location such as a Unity Catalog volume or external location. This provides both access control and governance, meeting the team's requirements.

Answer

Use the MLflow tracking server with a personal access token for each team member

Answer

Store experiment artifacts in a Unity Catalog volume and use workspace ACLs on the notebook

Answer

Enable inference tables on the MLflow experiment to log access

Question 7

A healthcare analytics team has a RAG application that must not reveal protected health information from other patients. They want to continuously monitor production traffic on their Mosaic AI Model Serving endpoint and alert when responses contain unsafe content. Which Databricks capability should they configure to evaluate each logged request and response against safety criteria and route flagged records for review?

Accepted Answer

Mosaic AI Agent Evaluation with a scheduled evaluation job over inference table data. Mosaic AI Agent Evaluation supplies judges for safety and other quality dimensions, and it can be scheduled against inference table data so every production interaction is scored. Records that fail the safety criteria can be written to a separate table for review, giving the team continuous, automated monitoring rather than manual spot checks.

Answer

Unity Catalog column masks on the inference table

Answer

MLflow autologging during model training

Answer

Delta Lake time travel on the inference table

Question 8

A RAG assistant built with Mosaic AI Agent Framework is in production. Product managers complain that answers are sometimes plausible but unsupported by the retrieved documents, and separately that the assistant sometimes ignores documents that clearly contain the answer. The team wants automated, scheduled checks in MLflow LLM Evaluation that separate these two failure modes so each can be triaged independently. Which TWO evaluation metrics should they add to the evaluation suite? (Choose two.)

Accepted Answer

Groundedness, which judges whether the claims in the generated response are supported by the retrieved context.. The two complaints map to distinct pipeline stages. A groundedness or faithfulness metric detects responses that are plausible but unsupported by the retrieved context, catching generation-side hallucination. A context recall metric detects whether retrieval supplied the evidence at all, catching retrieval-side gaps that make the model answer without support. Scoring both in the same scheduled evaluation run separates the failure modes so each can be fixed at its own stage, while latency and token metrics carry no information about either complaint.

Answer

P95 response time measured across all endpoint requests in the evaluation window.

Answer

Latency, measured as the wall-clock time from request receipt to final token of the response.

Answer

Total token count, computed as prompt tokens plus completion tokens per request.

Question 9

A financial services company has registered a Mosaic AI Agent application in Unity Catalog and wants to serve it through a Databricks Model Serving endpoint that queries a Delta table containing sensitive customer records. The security team requires that the endpoint access the table using a dedicated service principal with least privilege, and that the agent's LLM calls go through a governed gateway that logs usage. Which TWO configurations should the team apply to meet these requirements? (Choose two.)

Accepted Answer

Grant the serving endpoint's service principal SELECT on the specific customer table and its parent schema, and no broader catalog privileges.. The endpoint runs as its own service principal, so Unity Catalog grants on the specific table and schema give the agent exactly the read access it needs without broader exposure. Databricks AI Gateway on the serving endpoint provides governed routing, authentication, and logging of foundation-model calls. Together these two controls satisfy the least-privilege and governed-gateway requirements while keeping the agent's data access auditable.

Answer

Store the table's credentials in the agent's environment variables and have the agent open a direct JDBC connection to the Delta storage account.

Answer

Embed the customer records directly into the agent's MLflow model artifact so the endpoint never queries Unity Catalog at runtime.

Answer

Disable Unity Catalog enforcement on the endpoint so the agent can read the table with the workspace owner's inherited permissions.

Question 10

A healthcare analytics team deployed a RAG assistant on Databricks whose answers cite clinical policy documents. Compliance requires that every production answer be attributable to a specific retrieved chunk. During evaluation, the team notices the groundedness judge scores are high, but manual review finds answers that blend two policies into a statement neither document supports. Which change to their Mosaic AI Agent Evaluation configuration best detects this failure mode?

Accepted Answer

Enable the chunk-level citation assessment so each claim in the response is checked against the specific retrieved chunk it references, rather than scoring the response against the concatenated context.. The compliance requirement is per-answer attribution to a specific chunk, so the evaluation must verify each claim against the chunk it cites. Scoring the whole response against concatenated context allows a blended claim to pass whenever its components appear somewhere in the pool. Chunk-level citation assessment closes that gap by testing attribution directly, which is what distinguishes this failure from ordinary ungrounded hallucination.

Answer

Increase the number of retrieved chunks per query from five to twenty so the model has more supporting evidence available.

Answer

Raise the temperature setting on the served model to zero and re-run the evaluation suite to obtain more deterministic outputs.

Answer

Add a relevance judge that scores the retrieved chunks against the user question and filter out any chunk scoring below a fixed threshold.

Question 11

A Generative AI engineer is evaluating a RAG pipeline using MLflow LLM Evaluation on Databricks. They want to assess both the retrieval quality and the generation quality. Which TWO built-in evaluation metrics should they use? (Choose two.)

Accepted Answer

`relevance`. For a RAG pipeline, retrieval quality is assessed by relevance, which measures how well retrieved documents match the query. Generation quality is assessed by groundedness, which checks if the answer is supported by the retrieved context. Both are built-in metrics in MLflow LLM Evaluation and are specifically designed for RAG evaluation on Databricks.

Answer

`perplexity`

Answer

`toxicity`

Answer

`ROUGE`

Question 12

A GenAI engineer notices that a RAG agent's retrieval stage is returning relevant chunks, but the generated answers frequently omit key facts present in those chunks. The team wants a single evaluation metric that isolates whether the generator is using the provided context. Which metric should they focus on?

Accepted Answer

Groundedness, which checks whether claims in the answer are supported by the retrieved context.. Groundedness is the metric that inspects whether the answer's claims are supported by the retrieved context. Since the retriever is already delivering relevant chunks, a low groundedness score points squarely at the generator ignoring or misusing that evidence, giving the team a precise diagnostic for the stage that is failing.

Answer

Answer correctness against a reference answer.

Answer

p95 latency of the serving endpoint.

Answer

Context recall of the retriever.

Question 13

A Generative AI engineer is designing an evaluation harness for a customer-support RAG agent on Databricks. They need metrics that specifically assess the RETRIEVAL stage rather than the generation stage. (Choose two.)

Accepted Answer

Context recall, measuring whether the retrieved passages collectively contain the information present in the ground-truth answer.. Context recall and context precision are computed from the retrieved passages and ground-truth relevance, so they quantify what the retriever returned and in what order. Answer relevance, groundedness, and toxicity all depend on the generated response, which means they blend generation behavior into the score. Selecting the two context-based metrics keeps the harness focused on retrieval.

Answer

Answer relevance, measuring how well the final generated response addresses the user's original question.

Answer

Toxicity, measuring whether the generated response contains harmful or offensive content.

Answer

Groundedness, measuring whether the answer's claims are supported by the retrieved context.

Question 14

A GenAI team at a retail bank runs a RAG assistant on a Databricks Mosaic AI Model Serving endpoint. During a pilot, they captured end-user thumbs-up/down feedback in a Delta table but never joined it to the trace payloads. Six weeks later, hallucination complaints spike, yet the aggregate thumbs-down rate is unchanged. Which approach best resolves this discrepancy using Databricks-native tooling?

Accepted Answer

Join the Delta feedback table to MLflow traces on request_id and segment quality metrics by user cohort, prompt template version, and retrieved document source.. Aggregate feedback rates are diluted by cohort mix, so an unchanged overall thumbs-down percentage can coexist with a severe regression inside one prompt template version, document source, or user cohort. Correlating the stored human feedback with MLflow traces on request_id and then slicing by those dimensions surfaces where the hallucination spike actually lives, which is the prerequisite for any targeted fix.

Answer

Increase the endpoint's provisioned concurrency and enable autoscaling so responses are returned faster during peak hours.

Answer

Switch the judge from a smaller open model to a larger frontier model and re-run the nightly evaluation job over the last 30 days.

Answer

Retrain the embedding model on the latest product documentation and redeploy the index to the Vector Search endpoint.

Question 15

A team uses an LLM judge to score their RAG agent on Databricks and sees scores that fluctuate by several points between identical evaluation runs. They need more stable, reproducible quality signals for release gating. Which action best addresses the root cause?

Accepted Answer

Pin the judge model version and set its temperature to zero, then validate the judge against a human-labeled sample.. Fluctuating judge scores usually stem from nondeterministic sampling and drifting judge model versions. Pinning the judge version and setting temperature to zero makes each scoring pass reproducible, and validating the judge against human labels confirms that the stabilized scores are also accurate enough to gate releases reliably.

Answer

Increase the number of evaluation examples until the average stabilizes across runs.

Answer

Average the scores from three different judge models and use the mean for gating.

Answer

Switch from an LLM judge to BLEU or ROUGE string-overlap metrics.