Databricks-GenAI-Assoc Practice Test 2 — 10 Questions

Question 1

When evaluating LLM outputs, which TWO metrics are most appropriate for measuring the 'quality' of a response in a RAG system? (Choose two)

Accepted Answer

Faithfulness. Answering quality in RAG requires looking at two distinct dimensions: the correctness of the content (faithfulness) and the relevance of the answer to the user's intent. Faithfulness ensures the model remains grounded in the provided evidence, while answer relevance captures whether the model actually addressed the specific user question. These metrics provide a balanced view, ensuring the system is both accurate and useful, which is essential for end-user satisfaction.

Answer

Inference Latency

Answer

GPU Utilization

Answer

Training Iterations

Question 2

A developer needs to deploy a custom Python model that requires non-standard library dependencies. Which MLflow feature should the developer use to specify these environment requirements during model logging?

Accepted Answer

MLflow environment requirements (pip_requirements). When logging a custom model with MLflow, the developer should use the `pip_requirements` or `conda_env` parameter in the `mlflow.pyfunc.log_model` function. This ensures that the environment is correctly packaged and reproducible. During deployment, the Databricks Model Serving service reads these requirements to recreate the identical software environment, preventing runtime errors caused by missing dependencies when the model is loaded and served in the production environment.

Answer

Model Schema definition

Answer

Global workspace library settings

Answer

Custom model signatures

Question 3

Refer to the exhibit. An engineer wants to perform a canary deployment by routing 10% of traffic to a new version (version 6). How should the JSON traffic configuration be modified?

Accepted Answer

Add a new route for version 6 and set its traffic_percentage to 10, then set version 5 to 90.. Canary deployments involve splitting traffic between an established model version and a new candidate. To achieve this, the 'routes' list must include both versions, with the 'traffic_percentage' values summing to 100%. This controlled rollout allows teams to monitor the performance and accuracy of the new model on a small segment of production traffic before committing the full load to the new version.

Answer

Set the traffic_percentage of version 6 to 10 and keep version 5 at 100.

Answer

Remove version 5 and set version 6 to 10.

Answer

Create a new endpoint entirely for version 6 and switch the DNS record.

Question 4

A data scientist is using Databricks to fine-tune a large language model. The training data is stored in a Unity Catalog table that contains sensitive customer information. The data scientist needs to read the table but should not be able to see the raw values of certain columns. Which Unity Catalog feature should be used to dynamically mask the sensitive columns based on the user's group membership?

Accepted Answer

Dynamic column masking. Dynamic column masking in Unity Catalog enables the data scientist to see masked values for sensitive columns based on their group membership. By applying a column mask that checks group membership, the raw values are hidden from unauthorized users while still allowing the data scientist to read the table for fine-tuning.

Answer

Column-level encryption

Answer

Attribute-based access control

Answer

Row-level security

Question 5

When evaluating the performance of a RAG application, which metric is most useful for measuring the quality of the retrieved context?

Accepted Answer

Context Precision.. Context Precision measures how much of the retrieved information is actually relevant to the user query. This is a critical metric for RAG systems because high retrieval precision directly reduces the noise in the LLM input, leading to more accurate and focused answers. Measuring this allows engineers to iteratively improve the retrieval pipeline, which is essential for maximizing the utility of the application.

Answer

Total number of documents retrieved.

Answer

The total latency of the RAG pipeline.

Answer

The number of parameters in the LLM.

Question 6

Refer to the exhibit. An engineer is configuring a canary deployment for a churn prediction model. Based on the provided traffic configuration, what is the expected behavior of the endpoint?

Accepted Answer

90% of requests are routed to churn_v1 and 10% to churn_v2.. The configuration implements a traffic-splitting strategy, directing 90% of requests to 'churn_v1' and 10% to 'churn_v2'. This is a standard pattern for A/B testing or canary releases, allowing teams to validate new models in production with minimal risk. By monitoring the performance of the 10% traffic slice, engineers can decide whether to promote the model, ensuring stability before a full rollout.

Answer

The endpoint will error because total traffic must equal 100% for each model version individually.

Answer

The endpoint will only route traffic to churn_v1 after churn_v2 reaches capacity.

Answer

The endpoint will round-robin requests between v1 and v2 regardless of the weights.

Question 7

Which CI/CD approach for model deployment best minimizes downtime during a model update?

Accepted Answer

Use a blue-green deployment pattern.. A blue-green deployment strategy is the gold standard for minimizing downtime. By spinning up a new version of the model (green) alongside the existing one (blue) and switching traffic only after verifying the health of the green endpoint, engineers ensure zero downtime and an immediate rollback path if issues occur. This approach is critical for high-availability AI services where any interruption would negatively impact users.

Answer

Delete the old model and immediately deploy the new version.

Answer

Deploy all updates directly to the production endpoint.

Answer

Schedule deployments during peak traffic hours.

Question 8

When evaluating a generative AI model using the Mosaic AI Model Evaluation tool, what is the primary purpose of providing a 'baseline' dataset?

Accepted Answer

To provide a reference for comparative performance analysis.. A baseline dataset serves as a ground-truth reference point. By comparing the model's current outputs against this known high-quality set, engineers can objectively measure performance degradation or improvement. This benchmarking approach is fundamental to scientific model evaluation, ensuring that updates to the model or data retrieval logic do not silently introduce regressions in the quality of the generated answers during the deployment process.

Answer

To increase the number of tokens the model can generate.

Answer

To act as a secondary training set for fine-tuning.

Answer

To optimize the vector database search latency.

Question 9

Which TWO of the following are mandatory requirements for developing an AI application using the Databricks Mosaic AI Model Serving environment?

Accepted Answer

The model must be registered as a model version within a Unity Catalog schema.. Mosaic AI Model Serving requires specific configurations to ensure secure and performant access. Firstly, the model must be registered in Unity Catalog to maintain governance and lineage. Secondly, the serving endpoint requires defined compute resources, typically GPU-accelerated for LLMs, to handle inference requests. These requirements are essential for productionizing models, as they ensure that models are discoverable, governed, and have the necessary hardware to meet low-latency performance targets in real-time scenarios.

Answer

The model must be stored in a legacy Databricks Workspace folder.

Answer

The model must be served using a shared-access mode interactive cluster.

Answer

The application code must perform manual model sharding across nodes.

Question 10

An engineer is designing a Databricks RAG application that must support multi-turn conversations where follow-up questions refer to earlier turns. They want the retrieval step to remain accurate as the conversation progresses. Which TWO design elements should they include? (Choose two.)

Accepted Answer

Maintain a conversation history buffer and condense prior turns into a standalone query before invoking Vector Search.. Conversational RAG needs two coordinated elements: a rewritten, standalone query for retrieval, and conversation history for generation. Rewriting resolves references so Vector Search returns on-topic chunks, while including prior turns helps the model produce coherent follow-ups. Simply enlarging the retrieval window or removing filters does not address the ambiguity that multi-turn questions introduce.

Answer

Increase the number of retrieved chunks for every turn to compensate for conversational ambiguity.

Answer

Store the conversation history in the prompt and pass it unchanged to the retriever as the search query.

Answer

Disable metadata filtering during multi-turn conversations so more historical documents are eligible.