Databricks-GenAI-Assoc Practice Test 20 — 15 Questions

Question 1

An engineer uses MLflow LLM Evaluation with mlflow.evaluate() to score a RAG application. The judge model is configured with a temperature of 0.9 and no explicit metric thresholds are set. Reruns of the identical evaluation dataset produce relevance scores that swing by up to 20 percentage points, and the team cannot tell whether a prompt change helped. Which change most directly improves the reliability of the evaluation comparison?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the judge model's temperature to 0 and define fixed pass/fail thresholds for each metric before comparing runs.. Score swings across identical runs are caused by nondeterministic judge sampling, so the fix must remove that randomness and make the decision rule stable. Setting the judge temperature to zero makes each judgment reproducible, and predefined thresholds turn noisy continuous scores into consistent pass/fail outcomes. Enlarging the judge, duplicating rows, or hiding per-row results leaves the stochasticity in place and keeps run-to-run comparisons unreliable.

Answer

Increase the evaluation dataset size by duplicating each question three times so variance averages out across more judge calls.

Answer

Switch the evaluation to a larger judge model with more parameters while keeping the existing sampling settings.

Answer

Log only the aggregate mean relevance score per run and discard the per-row judgments from the MLflow run artifacts.

Question 2

An engineer is preparing to deploy a Mosaic AI Agent application with Databricks Asset Bundles. The bundle defines the agent, the serving endpoint, and a job that refreshes the vector index. The engineer wants the deployment to target a staging workspace and a production workspace with different endpoint names and different Unity Catalog catalog names, without editing files between deployments. Which approach should the engineer use?

Accepted Answer

Define bundle variables for the endpoint name and catalog name, and set their values per target in the bundle's databricks.yml targets section.. Databricks Asset Bundles let a single declarative project target multiple workspaces through targets, and variables can be given different values per target. Defining variables for the endpoint and catalog names and overriding them in each target lets the same bundle deploy correctly to staging and production without editing files. This preserves one source of truth and matches the scenario's requirement.

Answer

Hard-code the production endpoint and catalog names and use a post-deployment notebook to rename resources in staging.

Answer

Maintain two separate bundle projects, one per workspace, and keep them synchronized manually.

Answer

Store the environment-specific values in a secrets scope and read them at runtime inside the agent code.

Question 3

A GenAI engineer is developing a RAG application that uses a Vector Search index. The index is created from a Delta table that contains sensitive information. The security team requires that the engineer can audit which users have queried the index and what data they retrieved. Which Unity Catalog feature should the engineer enable to capture this information?

Accepted Answer

Audit logs in system tables. Audit logs in system tables capture all access and query events, including those against Vector Search indexes. They record the user, action, and timestamp, enabling the engineer to audit who queried the index and when. This is the appropriate Unity Catalog feature for meeting the security team's auditing requirement.

Answer

Unity Catalog lineage

Answer

Vector Search index metadata

Answer

Inference tables on the Vector Search endpoint

Question 4

A team deploys a Mosaic AI Agent application to a Databricks Model Serving endpoint. During load testing they observe that the first request after an idle period takes several seconds, while subsequent requests are fast. They want to eliminate this cold-start penalty for a latency-sensitive customer-facing application while keeping costs reasonable during off-peak hours. Which configuration should they apply?

Accepted Answer

Set the endpoint's minimum provisioned concurrency to a value greater than zero so at least one instance is always warm.. Cold starts occur when the endpoint has scaled down and must load a fresh model container on the next request. Provisioned concurrency with a minimum above zero keeps at least one instance warm at all times, removing the first-request latency penalty while still permitting scale-up under load and scale-down to the configured minimum during off-peak hours. Concurrency, inference tables, and image size do not guarantee warm capacity.

Answer

Reduce the agent's dependency footprint and re-log the model so the container image is smaller.

Answer

Enable inference tables on the endpoint to log requests and responses, which keeps the model warm.

Answer

Increase the endpoint's maximum concurrency per instance so a single warm instance can absorb all traffic.

Question 5

A support team operates a Databricks-hosted RAG assistant and wants end-user feedback to feed their monitoring dashboards. They plan to add a thumbs-up and thumbs-down control to the chat UI and log each vote alongside the request ID. What is the primary value of collecting this feedback for the evaluation and monitoring workflow?

Accepted Answer

It provides real-world signal that can be joined to traces for triage and dataset curation. End-user thumbs votes are weak but real labels tied to specific production requests. Joined with traces and request IDs, they drive triage of failing interactions and curation of new offline evaluation examples, connecting production monitoring to the evaluation dataset. They do not replace datasets, train models automatically, or improve metrics on their own.

Answer

It guarantees that groundedness scores will improve over time

Answer

It replaces the need for any offline evaluation dataset

Answer

It automatically fine-tunes the underlying foundation model

Question 6

A GenAI engineer monitors a customer-facing RAG assistant hosted on Databricks. After a routine re-indexing job, groundedness scores from the MLflow LLM Evaluation job drop sharply while answer relevance stays flat. The application prompt and the LLM serving endpoint were untouched. Which conclusion is best supported by these signals?

Accepted Answer

The re-indexing job degraded retrieval, so the generator receives weaker supporting context. The drop begins exactly when the index was rebuilt, and relevance remains flat, which isolates the change to the evidence supplied to the generator rather than to the prompt, the LLM, or the judge. Degraded retrieval after re-indexing, from altered chunking, embedding mismatches, or an incomplete build, is the best-supported conclusion and the right place to investigate first.

Answer

The LLM judge model was silently upgraded and is scoring more harshly

Answer

Users are asking harder questions than before, lowering groundedness

Answer

The generation prompt has drifted and needs to be rewritten

Question 7

A team has developed a GenAI chatbot using the Databricks Mosaic AI Agent Framework and logged it with MLflow. They now want to expose it as a REST API that their customer support portal can call, with autoscaling and built-in monitoring. Which Databricks capability should they use to host the agent?

Accepted Answer

Databricks Model Serving, deploying the MLflow-logged agent to a serving endpoint.. Mosaic AI Agent Framework applications logged with MLflow are served on Databricks Model Serving, which provides a REST endpoint, autoscaling, concurrency management, and built-in monitoring. That makes it the correct host for a customer support portal that needs to call the chatbot over HTTP. The other options are batch, development, or analytics compute surfaces that cannot serve interactive inference.

Answer

A Databricks SQL warehouse that exposes the agent through a SQL function the portal calls.

Answer

An all-purpose interactive cluster that the portal connects to over JDBC to invoke the agent.

Answer

A Databricks job that runs the agent on a schedule and writes responses to a Delta table the portal polls.

Question 8

A logistics company monitors a Databricks-hosted RAG assistant that answers questions about shipping regulations. Over one week, the retrieval index was rebuilt after a document refresh, and the team observes that the rate of answers flagged as unsupported by retrieved context rose sharply while retrieval relevance scores stayed flat. Which metric should they inspect first to determine whether the regression originates in the retrieval stage or the generation stage?

Accepted Answer

Context recall, measured against a labeled set of questions with known ground-truth source documents.. Context recall against a labeled ground-truth set directly tests whether the retriever brought back the documents that contain the answer. Because relevance measures topical similarity and can stay high even when authoritative chunks are missed, only recall can reveal that the rebuilt index now returns plausible but non-supporting passages, which would explain unsupported answers without any change in generation behavior.

Answer

The number of thumbs-up reactions collected from end users during the same week.

Answer

Endpoint request latency, comparing the p95 before and after the index rebuild.

Answer

Token usage per response, broken down by prompt and completion tokens on the serving endpoint.

Question 9

A media company uses an LLM-as-a-judge evaluation pipeline on Databricks to score a summarization assistant nightly. The judge is the same model family as the assistant and was given a rubric that rewards stylistic fluency. Over two months, nightly scores climb steadily while editor spot-checks find summaries increasingly omit key facts. Which corrective action best restores the evaluation's ability to detect this regression?

Accepted Answer

Revise the judge rubric to weight factual coverage against the source document, add a small human-labeled calibration set, and periodically verify judge agreement with editors.. The rising scores reflect a judge that rewards fluency because it shares a model family with the assistant and was given a style-oriented rubric, so the instrument itself is blind to factual omission. Restoring detection requires redefining the rubric around factual coverage of the source, anchoring it with human-labeled examples, and monitoring judge-human agreement so drift in the judge is caught before it masks another regression.

Answer

Increase the judge's temperature so its scores vary more between runs and regressions become statistically visible.

Answer

Run the judge more frequently, moving from nightly to hourly scoring, so trends are captured with finer granularity.

Answer

Switch the judge to the largest available frontier model and remove the rubric so the judge can apply its own judgment freely.

Question 10

A team uses MLflow LLM Evaluation with an LLM judge to score a summarization agent. On a 400-row golden dataset the judge marks 96% of summaries as relevant. Spot-checking reveals the judge approves nearly every summary whenever the summary is fluent, even when key facts are missing. The team wants a defensible quality signal before approving a release. Which action best addresses this judge weakness?

Accepted Answer

Add a small human-labeled calibration set and compare the judge's verdicts against human labels before trusting the automated relevance scores.. An automated judge that approves nearly every fluent summary is a biased measurement instrument, and the fix is to calibrate it rather than to change its sampling or the dataset size. Human-labeling a representative subset and measuring judge-human agreement exposes the leniency and tells the team how much to trust the scores. Only then can thresholds be set or a complementary fact-coverage check be added, producing a release signal that can be defended.

Answer

Replace the relevance metric with a cheaper lexical overlap score such as ROUGE against the reference summaries.

Answer

Expand the golden dataset from 400 rows to 4,000 rows and rerun the same relevance metric.

Answer

Raise the judge's temperature so it explores a wider range of judgments and produces a more nuanced relevance distribution.

Question 11

Which component of a Databricks Asset Bundle (DAB) allows you to define different configurations (e.g., instance sizes, variables) for development versus production environments?

Accepted Answer

The 'targets' block. The 'targets' block in the 'databricks.yml' file allows developers to define environment-specific configurations. This enables a single bundle to adapt to different workspaces or compute needs. By using targets, you can ensure that development deployments are lightweight and cost-effective, while production deployments are scaled for high availability and performance, all while keeping the underlying application code identical across all deployment stages.

Answer

The 'variables' block at the top level

Answer

The 'resources' block

Answer

The 'include' directive

Question 12

A data engineering team is using Databricks to prepare data for a RAG application. They want to ensure that document chunks are of consistent size and quality. Which tool should they use within the Databricks notebook environment to achieve this?

Accepted Answer

Distributed Spark UDFs with LangChain text splitters.. Databricks notebooks support a wide range of libraries, but for text processing, using Spark-native functions combined with libraries like LangChain allows for scalable and efficient chunking. By utilizing Spark's distributed processing capabilities, developers can process millions of documents in parallel. This is crucial for maintaining data quality in large-scale RAG systems, as it ensures that the context window of the LLM is not exceeded and that retrieval is focused and accurate.

Answer

A standard Python for-loop on a single worker node.

Answer

Manual data extraction into a local CSV file.

Answer

Databricks SQL commands to manually truncate strings.

Question 13

Refer to the exhibit. A developer is registering a model. Why is the model signature, as shown in the exhibit, considered a best practice for model registration?

Accepted Answer

It provides automated input validation to improve service reliability.. Model signatures define the schema for inputs and outputs, allowing the serving layer to perform automated validation. When an incoming request doesn't match the signature, the serving infrastructure can reject it immediately, providing useful error messages. This prevents invalid data from reaching the model, reducing runtime errors and debugging time, and ensures that the application behaves predictably in production environments, which is critical for maintaining high service quality and reliability.

Answer

It enables automatic data masking for sensitive strings.

Answer

It automatically scales the number of serving nodes based on the prompt size.

Answer

It forces the model to use a specific version of Python for execution.

Question 14

In a RAG application, which architectural component ensures that the system handles changes in the source data effectively?

Accepted Answer

The Delta Lake Change Data Feed (CDF).. The Delta Lake Change Data Feed (CDF) or the automatic synchronization of Vector Search indexes is designed specifically to track and apply changes (inserts, updates, deletes) to downstream systems. This ensures that the vector representation of the data is always in sync with the source of truth, avoiding the need for expensive full re-indexes and ensuring that retrieved information is always up-to-date.

Answer

The LLM's internal knowledge base.

Answer

The prompt engineering template.

Answer

The user session cache.

Question 15

Which approach is most effective for managing the dependencies of a custom ML model when deploying it to Mosaic AI Model Serving?

Accepted Answer

Include a 'requirements.txt' or 'conda.yaml' file in the model artifact.. Defining a custom environment using Conda or a 'requirements.txt' file ensures that the exact library versions required by the model are captured and recreated in the serving environment. This eliminates 'dependency hell' where a model works in development but fails in production due to library version mismatches. Ensuring consistent environments is a foundational step in robust ML engineering that guarantees reproducible performance in the serving infrastructure.

Answer

Install all dependencies manually on the serving cluster after deployment.

Answer

Assume the serving environment has all standard ML libraries pre-installed.

Answer

Package the entire Python environment inside the model artifact as a zip.