Databricks-GenAI-Assoc Application Development Practice Test 5 — 15 Questions

Question 1

A developer is building a RAG application using Mosaic AI Model Serving. They need to ensure that the embedding model endpoint is strictly accessed only by specific service principals within the workspace. Which feature should the developer configure to enforce this security requirement?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Model Serving Permissions. To restrict access to Mosaic AI Model Serving endpoints, developers should use Model Serving Permissions. By navigating to the Permissions tab of the specific endpoint, they can grant 'Can query' access exclusively to authorized service principals. This ensures that only authenticated and authorized applications can retrieve embeddings, protecting the model from unauthorized inference calls while maintaining a secure development lifecycle in Databricks.

Answer

Network Security Groups

Answer

Unity Catalog External Locations

Answer

Cluster Access Control Lists

Question 2

A team is deploying a LLM-based application using Databricks Model Serving. They want to implement robust observability and monitoring for their endpoint. Which TWO features should they utilize to track performance and quality metrics? (Select TWO)

Accepted Answer

Model Inference Tables. To effectively monitor a LLM application, teams must integrate both system-level performance metrics and application-level quality telemetry. Inference tables automatically log input and output data for analysis, while the native 'Monitoring' tab in the Model Serving UI provides latency and throughput metrics. Combined, these features provide a comprehensive view of how the model is performing, identifying bottlenecks in latency or degradation in response quality over time.

Answer

Unity Catalog Lineage

Answer

Databricks Delta Sharing

Answer

Workspace-level Audit Logs

Question 3

Refer to the exhibit. A developer is deploying a model using the provided JSON configuration. What is the primary benefit of setting 'scale_to_zero_enabled' to true in this production RAG application?

Accepted Answer

It reduces operational costs by shutting down compute resources when the endpoint is not serving traffic.. Setting 'scale_to_zero_enabled' to true allows the Databricks Model Serving endpoint to automatically shut down compute resources when no requests are being processed. This is highly beneficial for cost optimization, as it eliminates idle runtime costs during periods of inactivity. When a new request arrives, the service automatically initializes the model, ensuring cost efficiency without requiring manual intervention to scale the underlying infrastructure up or down.

Answer

It increases the throughput of the model by optimizing memory allocation during inference.

Answer

It ensures that the model always stays warm to minimize latency for every user request.

Answer

It enables high availability by automatically replicating the model across multiple availability zones.

Question 4

Refer to the exhibit. A developer encounters this error when trying to call a Model Serving endpoint from a job. Which action should the developer take to resolve this authorization failure?

Accepted Answer

Grant 'Can query' permissions to the service principal in the endpoint settings.. The error indicates a lack of 'Can query' permission for the service principal or user attempting to access the endpoint. To resolve this, the developer must update the permissions for the specific endpoint within the Databricks workspace. By granting the required 'Can query' privilege to the service principal running the job, the application will regain authorization to perform inference against the model, satisfying the security policy requirements.

Answer

Increase the 'workload_size' in the model configuration.

Answer

Update the workspace token expiration policy.

Answer

Restart the serving endpoint to refresh security policies.

Question 5

When developing a Generative AI application in Databricks, which tool provides a collaborative environment for engineers to write code, visualize data, and document their experiments using Markdown?

Accepted Answer

Databricks Notebooks. Databricks Notebooks provide a versatile, collaborative environment designed for data science and AI development. They allow developers to combine code, SQL queries, visualizations, and rich text documentation (Markdown) in a single document. This makes them ideal for building GenAI applications, as engineers can document their model tuning processes, test retrieval logic, and share findings with stakeholders directly within the platform, facilitating a highly efficient and iterative development workflow.

Answer

Databricks File System (DBFS)

Answer

Unity Catalog Explorer

Answer

Databricks SQL Editor

Question 6

A developer needs to deploy a custom Python model that requires non-standard library dependencies. Which MLflow feature should the developer use to specify these environment requirements during model logging?

Accepted Answer

MLflow environment requirements (pip_requirements). When logging a custom model with MLflow, the developer should use the `pip_requirements` or `conda_env` parameter in the `mlflow.pyfunc.log_model` function. This ensures that the environment is correctly packaged and reproducible. During deployment, the Databricks Model Serving service reads these requirements to recreate the identical software environment, preventing runtime errors caused by missing dependencies when the model is loaded and served in the production environment.

Answer

Model Schema definition

Answer

Global workspace library settings

Answer

Custom model signatures

Question 7

A developer is configuring a RAG application and needs to ensure that the LLM response is based on specific, trusted document snippets. Which technique, when implemented correctly, helps mitigate hallucination by grounding the response in provided context?

Accepted Answer

Retrieval Augmented Generation (RAG). Retrieval Augmented Generation (RAG) is the primary technique for grounding LLM responses. By retrieving relevant, trusted documents from a vector store based on a user's query and injecting those documents into the LLM's prompt, the developer forces the model to synthesize an answer based on specific retrieved context rather than relying solely on its internal training data. This significantly reduces hallucinations and increases the accuracy and relevance of the generated responses.

Answer

Increasing the model's temperature parameter

Answer

Fine-tuning the base model on all internal documents

Answer

Using a larger foundation model without context injection

Question 8

A GenAI engineer is building a retrieval-augmented generation (RAG) application using Databricks Vector Search. During testing, they observe that for some queries, the retrieval step returns document chunks that are semantically similar but not actually relevant to the user's question, leading to poor answer quality. They want to improve retrieval precision without changing the embedding model. Which of the following approaches is most appropriate?

Accepted Answer

Enable hybrid search on the Vector Search index to combine keyword-based and vector similarity scores.. Hybrid search enhances retrieval precision by blending vector similarity with keyword matching, ensuring that chunks containing exact query terms are ranked higher. This directly addresses the problem of semantically similar but irrelevant results without altering the embedding model. The other options either do not tackle the precision issue or violate the constraint of not changing the embedding model.

Answer

Reduce the number of retrieved chunks (top_k) to only the single highest-scoring chunk.

Answer

Switch the embedding model to a larger, more powerful model to improve semantic representation.

Answer

Increase the chunk size of the documents to include more context per retrieved chunk.

Question 9

A GenAI engineer is building a chatbot using Databricks Foundation Model APIs. They need the chatbot to maintain conversation context across multiple user turns and also allow the use of custom tools like a weather API. Which approach should they take?

Accepted Answer

Use the chat completion endpoint with a messages array that includes the full conversation history and define tools in the request.. The chat completion endpoint with a messages array is designed for multi-turn conversations and supports tool definitions. By including the full history, the model can reference prior context, and by defining tools, the model can request external API calls. This is the standard pattern for building conversational agents with Databricks Foundation Model APIs.

Answer

Use the chat completion endpoint but only send the latest user message, relying on the model to remember previous turns.

Answer

Use the completions endpoint and concatenate all previous user inputs and model outputs into a single prompt string.

Answer

Use the embeddings endpoint to encode the conversation history and pass the embeddings to the model for context.

Question 10

A developer is using MLflow to track experiments for a RAG application. They want to log the retrieval step's parameters, such as the number of documents retrieved (k) and the embedding model used. Which MLflow API should they use?

Accepted Answer

mlflow.log_param(). MLflow parameters are meant for logging configuration settings that are constant for a run. The number of documents retrieved and the embedding model are such settings. mlflow.log_param() records them as key-value pairs, enabling easy comparison across runs in the MLflow UI. Metrics, artifacts, and tags serve different purposes and are not suitable for this use case.

Answer

mlflow.log_artifact()

Answer

mlflow.set_tag()

Answer

mlflow.log_metric()

Question 11

An AI engineer is deploying a RAG application using Databricks Model Serving. They need to ensure the endpoint can handle high traffic with low latency and automatically scale based on demand. Which configuration should they use?

Accepted Answer

Configure autoscaling with a minimum of 1 and maximum of 10 replicas, and set appropriate concurrency per replica.. Autoscaling with a minimum and maximum replica count allows the endpoint to scale out during high traffic and scale in during low traffic, optimizing both latency and cost. Setting appropriate concurrency per replica ensures efficient utilization. This configuration provides the elasticity required for high traffic with low latency, while avoiding the pitfalls of fixed provisioning or overly restrictive concurrency limits.

Answer

Enable scale-to-zero and set a maximum concurrency of 1.

Answer

Disable autoscaling and manually provision 20 replicas to handle peak load.

Answer

Use a single large replica with maximum concurrency set to 100.

Question 12

A developer is using Databricks AI Functions to extract structured information from a large set of customer reviews stored in a Delta table. They want to apply a prompt to each review and store the results in a new column. Which function should they use?

Accepted Answer

ai_query(). Databricks AI Functions provide SQL-native access to generative AI models. ai_query() is the general-purpose function that accepts a prompt and a model endpoint, making it ideal for custom extraction tasks. It can be used in a SELECT statement to process each row and write the output to a new column. Other functions like ai_summarize and ai_classify are specialized and do not offer the same flexibility for structured extraction.

Answer

ai_summarize()

Answer

ai_extract()

Answer

ai_classify()

Question 13

A GenAI engineer has built a retrieval-augmented generation (RAG) application using Databricks Vector Search and a Databricks-hosted LLM served via Mosaic AI Model Serving. Users report that responses are sometimes irrelevant or cite incorrect document passages. The engineer wants to systematically improve answer quality by identifying which retrieved chunks are actually being used by the LLM. Which approach should the engineer take to capture the relationship between retrieved context and the generated response for later evaluation?

Accepted Answer

Enable MLflow Tracing on the RAG chain to log each retrieval and generation step with inputs and outputs.. MLflow Tracing instruments each stage of a RAG pipeline, recording retrieved chunks and the LLM's output as linked spans. This gives the engineer the data needed to see which context was passed and potentially used, enabling targeted improvements and evaluation. Other options either change retrieval behavior or log only endpoint-level data, lacking the granular linkage required.

Answer

Configure the Model Serving endpoint to log all requests and responses to a Delta table for offline analysis.

Answer

Increase the Vector Search index's embedding dimension to improve semantic matching.

Answer

Add a reranker model after Vector Search to reorder retrieved chunks by relevance score.

Question 14

A GenAI engineer is building a retrieval-augmented generation (RAG) application using Databricks Vector Search. They notice that the retriever sometimes returns irrelevant chunks that hurt answer quality. They want to add a reranking step to improve the relevance of retrieved documents before passing them to the LLM. Which component should they add to their RAG pipeline?

Accepted Answer

A cross-encoder reranker. A cross-encoder reranker is specifically designed to reorder retrieved documents by jointly encoding the query and each document, producing a relevance score. This step filters out irrelevant chunks and improves the quality of context passed to the LLM. In Databricks, you can incorporate a reranker as a separate component in a LangChain or custom RAG pipeline, often using a model served via Mosaic AI Model Serving or an external endpoint.

Answer

A larger embedding model

Answer

A fine-tuned LLM

Answer

A BM25 retriever

Question 15

A GenAI engineer is building an agent with LangChain on Databricks. The agent must call a Unity Catalog function `catalog.schema.get_weather` to fetch current weather. They want the LLM to decide when to invoke this function. Which LangChain component should they use to expose the Unity Catalog function to the LLM?

Accepted Answer

Databricks Unity Catalog function as a LangChain tool. To let an LLM decide when to call a Unity Catalog function, the function must be presented as a LangChain tool. The `UCFunctionToolkit` from Databricks integrates with LangChain, automatically generating tool definitions from Unity Catalog functions. This enables the agent to invoke the function dynamically based on the conversation, which is essential for building responsive GenAI applications.

Answer

Databricks Vector Search retriever

Answer

MLflow pyfunc model wrapper

Answer

Databricks SQL Connector