Databricks-GenAI-Assoc Practice Exam 3 — 20 Questions

Question 1

Refer to the exhibit. An engineer observes an unexpected drop in "relevance" for the latest deployment. What is the most likely cause related to the evaluation process itself?

Accepted Answer

The evaluation dataset is misaligned with current production query trends.. A drop in relevance often suggests that the evaluation dataset (eval_set_v4) may have become outdated or misaligned with the current production use cases. If the user base or query patterns shift, the original evaluation set may no longer accurately reflect the actual performance requirements. Regularly refreshing the evaluation dataset to match current production trends is essential to ensure that metrics remain valid indicators of system performance.

Answer

The model was trained on too much data.

Answer

The inference cluster has too much compute capacity.

Answer

The faithfulness score increased too much.

Question 2

You need to ingest data from an external JSON source into a Delta table. The source schema is inconsistent. Which strategy is most effective for preparing this data?

Accepted Answer

Implement a bronze-silver medallion pattern.. The 'bronze-to-silver' pattern is the industry standard in Databricks for handling inconsistent data. By loading raw JSON into a 'Bronze' table with a schema of type 'string' (or a single JSON column), you preserve all information for audit. You then use 'Silver' tables to perform schema enforcement, data cleaning, and type casting. This separation of concerns allows for robust error handling and iterative refinement of the data cleaning logic.

Answer

Use inferSchema = true in the read configuration.

Answer

Define a rigid schema at the ingestion point.

Answer

Convert the JSON to CSV before loading into Delta.

Question 3

A healthcare company uses Databricks to build a RAG application over clinical notes stored in a Unity Catalog table. An auditor requires proof that only authorized personnel can view raw note text, while the RAG application must use embeddings generated from those notes. The team wants to avoid copying data outside Unity Catalog. Which approach best satisfies the auditor while preserving RAG functionality?

Accepted Answer

Create a column mask on the note text column that returns a redacted value for unauthorized users, and grant the RAG service principal access to a separate embeddings table derived from the notes.. A column mask on the sensitive note text column enforces redaction for unauthorized identities at query time, which is auditable. By having the RAG application consume a derived embeddings table instead of raw text, the application retains functionality while raw PHI remains protected. This keeps governance within Unity Catalog and satisfies the auditor's requirement.

Answer

Move the clinical notes to an external location outside Unity Catalog and grant access via cloud IAM roles.

Answer

Grant the RAG application's service principal SELECT on the table and rely on the application to redact note text in its responses.

Answer

Use a dynamic view that filters rows based on the user's group and grant the RAG service principal access to that view.

Question 4

A data engineer needs to prepare a Delta table of customer reviews for embedding generation. The reviews contain HTML tags, inconsistent whitespace, and mixed casing that hurt embedding quality. Which preparation step should be applied before generating embeddings?

Accepted Answer

Normalize the text by stripping HTML tags, collapsing whitespace, and applying consistent casing.. Embedding models encode the exact tokens they receive, so HTML markup, irregular whitespace, and inconsistent casing add noise that dilutes semantic signal. Normalizing the text before embedding removes that noise and produces vectors that better reflect meaning. Storage format and vector dimension do not change the input text, so they cannot solve the quality problem.

Answer

Apply a hash of the raw review text as the embedding input to guarantee deterministic vectors.

Answer

Increase the embedding vector dimension to capture the additional characters in the raw text.

Answer

Store the reviews in a Parquet file instead of Delta to improve text compression before embedding.

Question 5

A fintech company runs a customer-support RAG assistant on Databricks. Before promoting a new prompt template from staging to production, the ML team must demonstrate that the change does not regress answer quality. Which TWO evaluation practices should they apply in Mosaic AI Agent Evaluation to make this promotion decision defensible? (Choose two.)

Accepted Answer

Pin the evaluation dataset, judge model version, and retrieval configuration so both templates are scored under identical conditions and results are reproducible.. A defensible promotion gate requires a controlled comparison where the prompt template is the only variable, which means running both templates over the same dataset with the judge model and retrieval configuration pinned, and inspecting per-question deltas instead of relying on averages that can mask localized regressions. Together these practices attribute any score movement to the change under test and make the decision reproducible.

Answer

Deploy the candidate template to a small percentage of production traffic and rely on user thumbs feedback alone as the promotion signal.

Answer

Increase the judge model size until its scores on the candidate template exceed the production template's scores.

Answer

Evaluate only the questions where the candidate template produced different answers, skipping the rest to save judge tokens.

Question 6

An engineer is packaging a GenAI agent application with Databricks Asset Bundles so that the same bundle deploys to a development and a production workspace. The agent's serving endpoint name must differ per target, and the production endpoint needs more concurrent capacity. Which mechanism in the bundle configuration should the engineer use?

Accepted Answer

Define target-specific variables in databricks.yml and reference them with ${var.endpoint_name} and ${var.min_replicas} in the resource definition.. Databricks Asset Bundles let you declare variables and override them per target, so a single resource definition can deploy the agent with different endpoint names and replica counts in development and production. Referencing variables with the ${var.} syntax keeps the bundle DRY while honoring environment differences. Hard-coded values, secret scopes, and post-deployment mutation scripts all fail to provide the clean, declarative per-target configuration the scenario requires.

Answer

Use a post-deployment notebook that calls the serving endpoints API to rename the endpoint and adjust replicas after each bundle deployment.

Answer

Store the endpoint name and replica count as secrets in a Databricks secret scope and read them at deployment time with the Databricks CLI.

Answer

Hard-code the production endpoint name and replica count, then maintain a separate copy of the bundle for development.

Question 7

When preparing a dataset for fine-tuning an LLM, you need to ensure the data is representative of the target domain. What is the most effective approach to detect and mitigate sampling bias in your training set using Databricks?

Accepted Answer

Perform statistical profiling on feature distributions and re-sample if necessary.. Using exploratory data analysis (EDA) with Databricks SQL or Spark to analyze distribution statistics is the most effective way to identify bias. By comparing the distribution of the training set to a representative sample of real-world production data, engineers can identify under-represented categories. This ensures that the fine-tuned model performs reliably across all expected input scenarios, preventing the model from developing blind spots.

Answer

Increase the batch size during the fine-tuning process.

Answer

Change the model architecture to a larger parameter count.

Answer

Use a random seed to shuffle the data before splitting.

Question 8

A developer is building a RAG application using Mosaic AI Model Serving. They need to ensure that the embedding model endpoint is strictly accessed only by specific service principals within the workspace. Which feature should the developer configure to enforce this security requirement?

Accepted Answer

Model Serving Permissions. To restrict access to Mosaic AI Model Serving endpoints, developers should use Model Serving Permissions. By navigating to the Permissions tab of the specific endpoint, they can grant 'Can query' access exclusively to authorized service principals. This ensures that only authenticated and authorized applications can retrieve embeddings, protecting the model from unauthorized inference calls while maintaining a secure development lifecycle in Databricks.

Answer

Network Security Groups

Answer

Unity Catalog External Locations

Answer

Cluster Access Control Lists

Question 9

A data scientist has developed a GenAI application locally and now needs to share it with stakeholders as an interactive web app inside the Databricks workspace. The app must use the workspace's existing authentication and compute. Which Databricks capability should be used?

Accepted Answer

Databricks Apps, which host the application within the workspace and integrate with workspace identity and compute.. Databricks Apps is designed for hosting interactive applications directly in the workspace, using the workspace's identity and compute. This lets stakeholders access the GenAI app through a URL with existing authentication, avoiding the need to build custom hosting or manage separate credentials.

Answer

A Databricks job with a notebook task scheduled to run every five minutes.

Answer

An MLflow experiment tracking run that logs the application as an artifact.

Answer

A Databricks notebook with the %run magic command to execute the application logic on demand.

Question 10

A team is using MLflow LLM Evaluation with the built-in answer_correctness metric to compare two prompt templates for a question-answering application. They notice that answer_correctness scores are nearly identical, but manual review shows one template produces answers that are factually correct yet omit key supporting details. Which additional built-in metric should they add to their evaluation to surface this difference?

Accepted Answer

answer_completeness. When answers are factually correct but differ in how much supporting detail they include, a completeness metric is required. answer_completeness compares the response against the ground truth to check whether all key points are covered, making it the right built-in metric to expose the omission of supporting details that answer_correctness alone misses.

Answer

groundedness

Answer

toxicity

Answer

answer_similarity

Question 11

A generative AI team is building a customer-support assistant on Databricks. The assistant must answer questions using the company's private knowledge base, and the team wants to minimize latency while ensuring that the LLM only uses retrieved documents. They plan to use Databricks Vector Search with a Delta table as the source. Which design choice best balances low latency and grounded responses?

Accepted Answer

Use a Vector Search index with a relevance score threshold and retrieve the top 5 chunks, then include those chunks in the prompt with instructions to answer only from the provided context.. Grounding requires retrieving only relevant context and instructing the model to use it. A Vector Search index with a relevance threshold and a small top-k (e.g., 5) reduces prompt size and latency while filtering weak matches. Including those chunks with an explicit instruction to answer from context limits hallucination. This combination balances performance and accuracy better than unfiltered or keyword-only retrieval.

Answer

Use a keyword-based search over the Delta table and pass the top 10 matching rows to the LLM, bypassing Vector Search entirely.

Answer

Store the entire knowledge base in a single prompt and use a long-context LLM to answer questions without retrieval.

Answer

Configure a Vector Search index with a small embedding dimension and retrieve the top 20 chunks, then pass all chunks directly to the LLM without filtering.

Question 12

An engineer is designing a RAG application that uses Databricks Vector Search to retrieve documents and a foundation model endpoint to generate answers. The team wants to log all user queries, retrieved documents, and generated responses for auditing and continuous improvement. They also need to monitor for drift in retrieval quality over time. Which Databricks capability should they integrate into the application design?

Accepted Answer

MLflow Tracing with inference tables on the model serving endpoint.. MLflow Tracing provides detailed spans across the RAG pipeline, capturing queries, retrieved documents, and LLM responses. Inference tables on the serving endpoint automatically log request and response payloads for auditing. This combination enables both observability and drift monitoring. Other options focus on data pipelines, dashboards, or governance, which do not capture the runtime application behavior needed for continuous improvement.

Answer

Delta Live Tables with expectations to validate the retrieved documents.

Answer

Databricks SQL dashboards querying the source Delta table for document updates.

Answer

Unity Catalog audit logs and lineage for the Vector Search index.

Question 13

A GenAI engineer has built a Retrieval Augmented Generation (RAG) application using Databricks Vector Search. The index is created on a Delta table that contains sensitive customer support transcripts. Company policy requires that end users can only retrieve chunks from documents they are authorized to view. The engineer wants to enforce this at query time without duplicating the index. Which approach should the engineer use?

Accepted Answer

Use Databricks Vector Search with the 'filter' parameter in the query, based on user attributes passed from the application.. Databricks Vector Search enables document-level security by allowing metadata filtering at query time. The engineer can include authorization attributes (such as group memberships) as metadata fields when creating the index, then pass a filter expression in the query that matches the user's attributes. This enforces access control without duplicating the index, aligning with the requirement to keep a single index while restricting retrieval to authorized documents.

Answer

Create a separate Vector Search index for each user group and route queries based on the user's group membership.

Answer

Apply Unity Catalog column masks on the embedding column of the source Delta table to redact sensitive text before vectorization.

Answer

Enable Attribute-Based Access Control (ABAC) on the source Delta table using row filter and column mask functions.

Question 14

A GenAI engineer has written a notebook that calls the Databricks Foundation Model APIs to summarize documents. Before the notebook can run in production, the security team wants to confirm exactly which workspace users and service principals are permitted to invoke the pay-per-token foundation model endpoints. Where should the engineer point them to review and manage those permissions?

Accepted Answer

The Unity Catalog privileges on the system.ai schema and its model functions. Foundation Model APIs are surfaced as Unity Catalog functions in the system.ai schema, so access is controlled with Unity Catalog privileges, chiefly EXECUTE on the relevant model function, plus the usual USE CATALOG and USE SCHEMA on system.ai. Directing the security team to those grants gives an auditable, centralized view of exactly which users and service principals can invoke each pay-per-token model.

Answer

The Unity Catalog metastore's default storage location settings

Answer

The workspace admin settings page for cluster policies

Answer

The Azure Databricks access connector assigned to the workspace

Question 15

A developer is building a RAG application and notices that the retrieval step often returns irrelevant context. Which step in the pipeline should be improved to address this?

Accepted Answer

Refine the chunking strategy and embedding model quality.. Improving retrieval accuracy often involves enhancing the quality of the embeddings or the text chunking strategy. By adjusting how data is partitioned into chunks (e.g., overlapping chunks, semantic chunking) or refining the embedding model, the vector search index can better capture the semantic meaning of the documents. This is a common and critical improvement path in RAG development, as the quality of the retrieval is fundamentally limited by the index's representational capability.

Answer

Increase the number of API calls to the LLM.

Answer

Enable auto-scaling on the serving endpoint.

Answer

Switch to a larger, more expensive LLM.

Question 16

Refer to the exhibit. An engineer is automating the deployment of a model to an endpoint using the Databricks CLI. Based on the error log provided, what is the most appropriate action to resolve this deployment failure?

Accepted Answer

Grant the service principal 'USE' and 'READ' permissions on the catalog, schema, and model.. Deployment failures due to 'INSUFFICIENT_PERMISSIONS' often stem from the service principal used by the CI/CD pipeline lacking the necessary grants on the registered model in Unity Catalog. The engineer must ensure the principal has 'USE CATALOG', 'USE SCHEMA', and 'READ' permissions on the model version. Addressing this at the Unity Catalog level is the standard procedure for cross-service authorization in Databricks.

Answer

Upgrade the model serving endpoint to a higher capacity cluster size.

Answer

Delete the model serving endpoint and recreate it using a personal access token.

Answer

Restart the Databricks workspace to force an update of the internal permission cache.

Question 17

A developer is configuring a RAG application and needs to ensure that the LLM response is based on specific, trusted document snippets. Which technique, when implemented correctly, helps mitigate hallucination by grounding the response in provided context?

Accepted Answer

Retrieval Augmented Generation (RAG). Retrieval Augmented Generation (RAG) is the primary technique for grounding LLM responses. By retrieving relevant, trusted documents from a vector store based on a user's query and injecting those documents into the LLM's prompt, the developer forces the model to synthesize an answer based on specific retrieved context rather than relying solely on its internal training data. This significantly reduces hallucinations and increases the accuracy and relevance of the generated responses.

Answer

Increasing the model's temperature parameter

Answer

Fine-tuning the base model on all internal documents

Answer

Using a larger foundation model without context injection

Question 18

Which TWO actions are necessary to ensure that a model serving endpoint in Databricks remains available and performant during peak traffic hours?

Accepted Answer

Enable auto-scaling for the serving endpoint.. Maintaining performance requires both vertical and horizontal scaling strategies. First, configuring auto-scaling on the endpoint allows the infrastructure to adjust to fluctuating demand automatically. Second, ensuring the endpoint uses appropriate instance types—such as GPU-accelerated instances for LLMs—ensures the compute capacity is sufficient for inference tasks. These two actions are foundational for achieving high availability and low latency, preventing request timeouts and ensuring a smooth experience during heavy usage periods.

Answer

Disable the serving endpoint's logging to save on compute cycles.

Answer

Configure the endpoint to run on a single, fixed-size node to minimize latency.

Answer

Manually restart the endpoint every hour to clear the cache.

Question 19

A data engineer is assembling a fine-tuning dataset from a Delta table of conversation transcripts. Each transcript contains a list of message objects with a role and a content field, and the training job requires one row per conversation with the messages serialized into the expected format. Which transformation should the engineer apply?

Accepted Answer

Use the collect_list aggregate to group the message objects back into an array per conversation and serialize that array into the required format.. The training job requires one row per conversation with the message sequence preserved, so the individual message rows must be grouped back together. collect_list aggregates messages into an ordered array per conversation, and serializing that array yields the nested format the job expects. Exploding, pivoting, or flattening would alter the structure and lose the required per-conversation layout.

Answer

Use the explode function to expand each message object into a separate row before writing the dataset.

Answer

Use the pivot function to turn message roles into columns with content values in the cells.

Answer

Use the flatten function to collapse the nested message array into a single string per row.

Question 20

Refer to the exhibit. An engineer is testing a model endpoint. The outputs are too brief and often stop mid-sentence. What is the most likely cause?

Accepted Answer

The max_new_tokens limit is too low for the expected response length.. The 'max_new_tokens' parameter is set to 512, which imposes a hard limit on the length of the generated output. If the model needs more tokens to complete its thought, it will be abruptly cut off. Increasing this limit is the logical troubleshooting step when encountering truncation issues in generative model responses, ensuring the model has sufficient space to finalize its output properly.

Answer

The temperature is too low for the model to generate full sentences.

Answer

The top_p parameter is too high, leading to early termination.

Answer

The model version is incompatible with the specified parameters.