Databricks-GenAI-Assoc Practice Test 5 — 10 Questions

Question 1

Refer to the exhibit. An engineer observes an unexpected drop in "relevance" for the latest deployment. What is the most likely cause related to the evaluation process itself?

Accepted Answer

The evaluation dataset is misaligned with current production query trends.. A drop in relevance often suggests that the evaluation dataset (eval_set_v4) may have become outdated or misaligned with the current production use cases. If the user base or query patterns shift, the original evaluation set may no longer accurately reflect the actual performance requirements. Regularly refreshing the evaluation dataset to match current production trends is essential to ensure that metrics remain valid indicators of system performance.

Answer

The model was trained on too much data.

Answer

The inference cluster has too much compute capacity.

Answer

The faithfulness score increased too much.

Question 2

You need to ingest data from an external JSON source into a Delta table. The source schema is inconsistent. Which strategy is most effective for preparing this data?

Accepted Answer

Implement a bronze-silver medallion pattern.. The 'bronze-to-silver' pattern is the industry standard in Databricks for handling inconsistent data. By loading raw JSON into a 'Bronze' table with a schema of type 'string' (or a single JSON column), you preserve all information for audit. You then use 'Silver' tables to perform schema enforcement, data cleaning, and type casting. This separation of concerns allows for robust error handling and iterative refinement of the data cleaning logic.

Answer

Use inferSchema = true in the read configuration.

Answer

Define a rigid schema at the ingestion point.

Answer

Convert the JSON to CSV before loading into Delta.

Question 3

A healthcare company uses Databricks to build a RAG application over clinical notes stored in a Unity Catalog table. An auditor requires proof that only authorized personnel can view raw note text, while the RAG application must use embeddings generated from those notes. The team wants to avoid copying data outside Unity Catalog. Which approach best satisfies the auditor while preserving RAG functionality?

Accepted Answer

Create a column mask on the note text column that returns a redacted value for unauthorized users, and grant the RAG service principal access to a separate embeddings table derived from the notes.. A column mask on the sensitive note text column enforces redaction for unauthorized identities at query time, which is auditable. By having the RAG application consume a derived embeddings table instead of raw text, the application retains functionality while raw PHI remains protected. This keeps governance within Unity Catalog and satisfies the auditor's requirement.

Answer

Move the clinical notes to an external location outside Unity Catalog and grant access via cloud IAM roles.

Answer

Grant the RAG application's service principal SELECT on the table and rely on the application to redact note text in its responses.

Answer

Use a dynamic view that filters rows based on the user's group and grant the RAG service principal access to that view.

Question 4

A data engineer needs to prepare a Delta table of customer reviews for embedding generation. The reviews contain HTML tags, inconsistent whitespace, and mixed casing that hurt embedding quality. Which preparation step should be applied before generating embeddings?

Accepted Answer

Normalize the text by stripping HTML tags, collapsing whitespace, and applying consistent casing.. Embedding models encode the exact tokens they receive, so HTML markup, irregular whitespace, and inconsistent casing add noise that dilutes semantic signal. Normalizing the text before embedding removes that noise and produces vectors that better reflect meaning. Storage format and vector dimension do not change the input text, so they cannot solve the quality problem.

Answer

Apply a hash of the raw review text as the embedding input to guarantee deterministic vectors.

Answer

Increase the embedding vector dimension to capture the additional characters in the raw text.

Answer

Store the reviews in a Parquet file instead of Delta to improve text compression before embedding.

Question 5

A fintech company runs a customer-support RAG assistant on Databricks. Before promoting a new prompt template from staging to production, the ML team must demonstrate that the change does not regress answer quality. Which TWO evaluation practices should they apply in Mosaic AI Agent Evaluation to make this promotion decision defensible? (Choose two.)

Accepted Answer

Pin the evaluation dataset, judge model version, and retrieval configuration so both templates are scored under identical conditions and results are reproducible.. A defensible promotion gate requires a controlled comparison where the prompt template is the only variable, which means running both templates over the same dataset with the judge model and retrieval configuration pinned, and inspecting per-question deltas instead of relying on averages that can mask localized regressions. Together these practices attribute any score movement to the change under test and make the decision reproducible.

Answer

Deploy the candidate template to a small percentage of production traffic and rely on user thumbs feedback alone as the promotion signal.

Answer

Increase the judge model size until its scores on the candidate template exceed the production template's scores.

Answer

Evaluate only the questions where the candidate template produced different answers, skipping the rest to save judge tokens.

Question 6

An engineer is packaging a GenAI agent application with Databricks Asset Bundles so that the same bundle deploys to a development and a production workspace. The agent's serving endpoint name must differ per target, and the production endpoint needs more concurrent capacity. Which mechanism in the bundle configuration should the engineer use?

Accepted Answer

Define target-specific variables in databricks.yml and reference them with ${var.endpoint_name} and ${var.min_replicas} in the resource definition.. Databricks Asset Bundles let you declare variables and override them per target, so a single resource definition can deploy the agent with different endpoint names and replica counts in development and production. Referencing variables with the ${var.} syntax keeps the bundle DRY while honoring environment differences. Hard-coded values, secret scopes, and post-deployment mutation scripts all fail to provide the clean, declarative per-target configuration the scenario requires.

Answer

Use a post-deployment notebook that calls the serving endpoints API to rename the endpoint and adjust replicas after each bundle deployment.

Answer

Store the endpoint name and replica count as secrets in a Databricks secret scope and read them at deployment time with the Databricks CLI.

Answer

Hard-code the production endpoint name and replica count, then maintain a separate copy of the bundle for development.

Question 7

When preparing a dataset for fine-tuning an LLM, you need to ensure the data is representative of the target domain. What is the most effective approach to detect and mitigate sampling bias in your training set using Databricks?

Accepted Answer

Perform statistical profiling on feature distributions and re-sample if necessary.. Using exploratory data analysis (EDA) with Databricks SQL or Spark to analyze distribution statistics is the most effective way to identify bias. By comparing the distribution of the training set to a representative sample of real-world production data, engineers can identify under-represented categories. This ensures that the fine-tuned model performs reliably across all expected input scenarios, preventing the model from developing blind spots.

Answer

Increase the batch size during the fine-tuning process.

Answer

Change the model architecture to a larger parameter count.

Answer

Use a random seed to shuffle the data before splitting.

Question 8

A developer is building a RAG application using Mosaic AI Model Serving. They need to ensure that the embedding model endpoint is strictly accessed only by specific service principals within the workspace. Which feature should the developer configure to enforce this security requirement?

Accepted Answer

Model Serving Permissions. To restrict access to Mosaic AI Model Serving endpoints, developers should use Model Serving Permissions. By navigating to the Permissions tab of the specific endpoint, they can grant 'Can query' access exclusively to authorized service principals. This ensures that only authenticated and authorized applications can retrieve embeddings, protecting the model from unauthorized inference calls while maintaining a secure development lifecycle in Databricks.

Answer

Network Security Groups

Answer

Unity Catalog External Locations

Answer

Cluster Access Control Lists

Question 9

A data scientist has developed a GenAI application locally and now needs to share it with stakeholders as an interactive web app inside the Databricks workspace. The app must use the workspace's existing authentication and compute. Which Databricks capability should be used?

Accepted Answer

Databricks Apps, which host the application within the workspace and integrate with workspace identity and compute.. Databricks Apps is designed for hosting interactive applications directly in the workspace, using the workspace's identity and compute. This lets stakeholders access the GenAI app through a URL with existing authentication, avoiding the need to build custom hosting or manage separate credentials.

Answer

A Databricks job with a notebook task scheduled to run every five minutes.

Answer

An MLflow experiment tracking run that logs the application as an artifact.

Answer

A Databricks notebook with the %run magic command to execute the application logic on demand.

Question 10

A team is using MLflow LLM Evaluation with the built-in answer_correctness metric to compare two prompt templates for a question-answering application. They notice that answer_correctness scores are nearly identical, but manual review shows one template produces answers that are factually correct yet omit key supporting details. Which additional built-in metric should they add to their evaluation to surface this difference?

Accepted Answer

answer_completeness. When answers are factually correct but differ in how much supporting detail they include, a completeness metric is required. answer_completeness compares the response against the ground truth to check whether all key points are covered, making it the right built-in metric to expose the omission of supporting details that answer_correctness alone misses.

Answer

groundedness

Answer

toxicity

Answer

answer_similarity