Databricks-GenAI-Assoc Practice Test 18 — 15 Questions

Question 1

A healthcare company uses Databricks to build a GenAI chatbot that answers questions from patient records stored in a Delta table. The records contain PHI, and the company must ensure that the chatbot never returns PHI to unauthorized users. The security team wants to enforce policies at the data layer so that even if the LLM is manipulated, it cannot access PHI. Which Unity Catalog feature should be used to dynamically redact PHI columns based on the user's group membership?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Column masks using a user-defined function that returns redacted values for unauthorized groups.. Unity Catalog column masks are designed to dynamically redact column values based on the user's identity or group membership. By applying a mask function to PHI columns, the healthcare company can ensure that unauthorized users see redacted values while authorized users see the actual data. This enforcement occurs at the data layer, preventing PHI leakage even if the LLM is manipulated, and it applies to all queries against the table.

Answer

Row-level security with a filter function that excludes rows containing PHI.

Answer

Attribute-based access control (ABAC) policies defined on the catalog to deny access to PHI columns.

Answer

Dynamic view that joins the patient records with a permissions table and filters out PHI columns.

Question 2

A data scientist wants to use a Unity Catalog registered model in a GenAI pipeline. The model was trained on sensitive data, and the governance team requires that the model's lineage back to the training dataset be traceable. Which Unity Catalog capability provides this traceability?

Accepted Answer

Unity Catalog lineage graph that automatically tracks data and model dependencies.. Unity Catalog lineage automatically tracks the flow of data through notebooks, jobs, and models, including the training dataset used to produce a model version. This gives the governance team an auditable, automated trace from model to data. Manual methods such as tags or comments are not reliable for compliance.

Answer

Model version tags that describe the training data.

Answer

Workspace notebook comments that document the training data path.

Answer

Model signature that records input and output schema.

Question 3

A team is designing a Databricks GenAI application that must return grounded answers with citations to source documents. The application uses Databricks Vector Search for retrieval and a Foundation Model API for generation. Which TWO design choices are required to return accurate citations alongside each answer? (Choose two.)

Accepted Answer

Instruct the model in the prompt to reference the provided chunks by their identifiers and include those identifiers in the response.. Citations require two things working together: source metadata attached to each retrieved chunk so the application knows where content came from, and prompt instructions that tell the model to attribute statements to specific chunk identifiers. Precision tuning, billing mode, and temperature settings do not supply the provenance information or the attribution behavior needed to render citations.

Answer

Set the model's temperature to zero to guarantee that citations are factually correct.

Answer

Increase the embedding model's dimensionality to the maximum supported value so retrieval is more precise.

Answer

Enable Foundation Model API pay-per-token billing so the model has access to citation formatting.

Question 4

A generative AI engineer registers a model in Unity Catalog and wants a downstream application to call it for inference without granting the application broad workspace access. The engineer is told to grant a specific Unity Catalog privilege on the registered model so the application can invoke it. Which privilege should be granted?

Accepted Answer

EXECUTE. Registered models in Unity Catalog are invoked under the EXECUTE privilege, which authorizes a principal to call the model without granting ownership or modification rights. Granting EXECUTE to the application's identity, alongside any needed USE CATALOG and USE SCHEMA to resolve the path, provides least-privilege inference access for the downstream application.

Answer

SELECT

Answer

MODIFY

Answer

USE CATALOG

Question 5

A GenAI application uses a Databricks Model Serving endpoint hosting a foundation model, and the team wants structured JSON output that conforms to a fixed schema for downstream parsing. Responses sometimes include prose or markdown fences that break the parser. Which design change is most likely to produce reliably parseable output?

Accepted Answer

Define the expected structure with a response format or structured-output parameter on the serving request so the endpoint constrains generation to the schema.. Declaring a response format or structured-output schema makes the endpoint constrain decoding so the generated text matches the required structure, removing prose and markdown fences by construction. Lowering temperature, raising the token limit, or stripping fences afterward only reduce or mask the symptom without guaranteeing conformance.

Answer

Add a regex-based post-processor that strips markdown fences and extracts the first balanced JSON object from the raw response.

Answer

Lower the temperature to zero and add the phrase 'return only JSON' to the end of the user message.

Answer

Increase max_tokens so the model has room to finish the JSON object without being truncated mid-structure.

Question 6

A GenAI engineer is designing a RAG application on Databricks that must support multi-turn conversations where users refer to earlier messages, and the application must keep responses grounded in retrieved documents. Which TWO design elements are required to meet these requirements? (Choose two.)

Accepted Answer

Rewrite the user query using conversation history before performing vector search retrieval.. Multi-turn conversation support requires passing relevant prior turns to the model so it can resolve references, and grounding requires that retrieval use a query rewritten with conversation context so the right documents are fetched. Together these elements ensure the model understands follow-up intent and answers from authoritative retrieved content. The other options either reduce determinism, add storage without context, or remove retrieval.

Answer

Increase the LLM temperature so the model can explore multiple interpretations of ambiguous follow-up questions.

Answer

Store all conversation turns in a Delta table and query that table at inference time for each user message.

Answer

Disable retrieval for follow-up questions and rely on the model's parametric knowledge.

Question 7

A platform team is serving a retrieval-augmented generation application. The vector index is built from a Delta table in Unity Catalog, and the team wants every similarity search executed by an end user to be automatically restricted to documents that user is allowed to see, without maintaining a separate index per department. Which approach best enforces this at query time?

Accepted Answer

Create a row filter function on the source Delta table that filters rows by the invoking user's group membership. Row filters evaluate the invoking user's identity as part of the query, so placing the filter on the source Delta table means every read, including retrieval reads that feed similarity search, is automatically scoped to allowed rows. This keeps a single governed index and avoids per-department duplication while still enforcing per-user document visibility at query time.

Answer

Apply a column mask to the embedding column so unauthorized users receive null vectors

Answer

Build one vector index per department and route queries based on the caller's group

Answer

Grant SELECT only to a service principal and have the application connect as that principal for all users

Question 8

A team runs an LLM-as-a-judge evaluation on Databricks using the built-in `mlflow.evaluate()` with `model_type="databricks-agent"`. They notice that the judge model, a serving endpoint, is producing scores that are consistently inflated compared to human review. Which configuration change should the team make first to improve the reliability of the evaluation?

Accepted Answer

Switch the judge to a different, stronger model endpoint and calibrate it using a small human-labeled dataset with known ground-truth scores.. LLM judges can exhibit systematic bias, such as consistently inflated scores. The most effective first step is to use a stronger judge model and calibrate it against human-labeled data with known scores. This improves reliability and aligns with Databricks recommendations for validating judges before using them in production monitoring. Simply changing temperature or filtering data does not address the root cause.

Answer

Reduce the number of evaluation examples to only those where the judge and human agree, then report the average score.

Answer

Increase the temperature parameter of the judge model to 1.0 to encourage more diverse scoring.

Answer

Replace the LLM judge with a BLEU score comparison against a reference answer for every question.

Question 9

A Databricks Generative AI engineer has deployed a RAG application and is now setting up production monitoring. They want to automatically detect when the distribution of incoming user questions diverges from the distribution seen during development, so they can trigger retraining or prompt adjustments. Which Databricks capability should they configure?

Accepted Answer

Inference tables with Lakehouse Monitoring for the endpoint's payload and response columns. Inference tables capture the actual request and response payloads served by the endpoint, and Lakehouse Monitoring can build profiles and drift metrics over those tables. Together they provide the automated distributional comparison needed to detect when live questions diverge from the development baseline, which is exactly the monitoring goal.

Answer

Mosaic AI Agent Evaluation with a custom LLM judge for every request

Answer

Delta Live Tables expectations on the raw question stream

Answer

MLflow Model Registry stage transitions with webhook notifications

Question 10

A Generative AI engineer is using MLflow Tracing to monitor a RAG application deployed on Mosaic AI Model Serving. They want to capture the retrieved documents, the final prompt, and the model's response for each request to debug a quality issue. Which approach should they use to ensure all three are logged in a single trace?

Accepted Answer

Instrument the application with the MLflow tracing SDK, creating spans for retrieval, prompt construction, and generation within the same trace context.. To capture retrieved documents, the final prompt, and the model response in a single trace, the engineer should use MLflow Tracing with manual instrumentation. Spans for each step share a trace context, providing a unified view. This is the native Databricks approach for debugging RAG applications and integrates with Mosaic AI Model Serving for production monitoring.

Answer

Use the `mlflow.evaluate()` API with a custom evaluator that logs retrieved documents, prompt, and response as metrics.

Answer

Enable inference table logging on the serving endpoint and query the payload column for the retrieved documents and response.

Answer

Configure the serving endpoint to log all requests to a Delta table and then join with a separate table of retrieved documents using a request ID.

Question 11

An engineer is designing a GenAI application that must call an external LLM provider's API. The provider key must not appear in notebook code, job logs, or Git. The team already uses Databricks and wants the key to be injected into the serving endpoint at runtime. Which Databricks capability should the engineer use?

Accepted Answer

Databricks secrets stored in a secret scope, referenced from the serving endpoint configuration using secret references.. Databricks secret scopes with secret references keep credentials encrypted and out of source control, logs, and notebooks, while still allowing Model Serving to inject them at runtime. Spark config values, Unity Catalog volume files, and model signature parameters all expose the key through metadata, logs, or request payloads and do not provide runtime injection for serving endpoints.

Answer

Hard-coding the provider key in the model signature so the endpoint receives it as a request parameter.

Answer

A Unity Catalog volume containing a text file with the provider key, mounted into the serving endpoint.

Answer

Environment variables defined in the cluster's Spark configuration and read by the notebook at runtime.

Question 12

A media company's RAG assistant answers questions about its streaming catalog. Users report that the assistant often returns answers that ignore the retrieved documents and instead rely on outdated information from the base model. The team wants an MLflow LLM Evaluation metric that measures how much of the final answer is actually derived from the retrieved context rather than the model's prior knowledge. Which metric best fits this need?

Accepted Answer

groundedness. Groundedness directly assesses whether the response is supported by the retrieved context, so a pattern of ignoring documents and using stale base-model knowledge will show up as low groundedness. That makes it the metric that quantifies the behavior the team is investigating in their catalog assistant.

Answer

relevance

Answer

context_sufficiency

Answer

answer_correctness

Question 13

A team runs a RAG chatbot whose MLflow evaluation with the built-in groundedness judge previously scored well. After they swap the retriever for a new embedding model, groundedness scores drop sharply even though the generator model and prompt are unchanged. They confirm the judge model itself is unchanged. Which action should they take FIRST to diagnose the regression?

Accepted Answer

Inspect the retrieved context chunks logged per evaluation row to verify whether the new retriever is returning passages that no longer support the generated claims.. Groundedness measures whether the answer is supported by the retrieved context, so a change to retrieval is the most likely culprit. Inspecting the logged retrieved chunks per evaluation row reveals whether the new embedding model returns less relevant or contradictory passages, which the judge then flags as unsupported claims. This isolates retrieval as the cause before any remediation.

Answer

Roll back the embedding model change immediately and re-run the evaluation to confirm the previous score returns.

Answer

Retrain the judge model on a fresh labeled dataset so its scoring distribution aligns with the new embedding model.

Answer

Increase the temperature of the generator model to encourage more diverse phrasing that the groundedness judge can score more reliably.

Question 14

A GenAI engineering team has deployed a customer-support RAG chain on Databricks and registered it in Unity Catalog. They now want MLflow 3 to automatically score every production request for groundedness and relevance without writing custom scoring code, and to persist those assessments against the logged traces. Which approach should they use?

Accepted Answer

Attach MLflow scorers to the deployed agent so that assessments are computed on live traces and stored in the MLflow experiment.. MLflow 3 production monitoring is designed for exactly this need: built-in scorers are attached to a deployed agent rather than invoked manually, and assessments are recorded on live traces inside the MLflow experiment. This removes custom scoring code, keeps quality signals tied to each request, and enables continuous monitoring without batch jobs or dashboard workarounds.

Answer

Configure the endpoint's autoscaling policy to collect evaluation metrics alongside throughput metrics.

Answer

Create a Databricks SQL dashboard that re-runs mlflow.evaluate() on the inference table every five minutes.

Answer

Enable the system table system.serving.endpoint_usage and query it for groundedness scores.

Question 15

A company is deploying a GenAI chatbot that uses a Databricks Model Serving endpoint hosting a fine-tuned Llama 2 model. The endpoint is registered in Unity Catalog. The security team wants to restrict which groups can invoke the endpoint and also log all inference requests for auditing. Which combination of Unity Catalog and Databricks features should the engineer use to meet these requirements?

Accepted Answer

Grant EXECUTE on the model version to the allowed groups and enable inference tables on the serving endpoint. To control who can invoke a model served by Databricks, you grant the EXECUTE privilege on the model version in Unity Catalog. Inference tables are a built-in feature of Model Serving that automatically log request and response data to a Delta table for auditing and monitoring. This combination directly addresses the security team's requirements for access restriction and request logging.

Answer

Grant USE CATALOG and USE SCHEMA on the model's parent schema and enable verbose logging on the endpoint

Answer

Grant SELECT on the model version to the allowed groups and enable audit logs in the workspace admin console

Answer

Use a personal access token (PAT) with scoped permissions and configure a Delta table as a sink for endpoint logs