Hard Difficulty Questions

Hardcode API credentials within the model inference script.

Why it fails: Hardcoding credentials is a severe security risk that exposes sensitive information to anyone with read access to the codebase. Instead, secrets management via Databricks Secrets or Unity Catalog should be used to securely inject credentials at runtime, maintaining a strict separation between code and sensitive configuration data.

Disable automatic scaling to maintain consistent latency.

Why it fails: Disabling automatic scaling does not improve observability and often leads to performance bottlenecks during traffic spikes. Proper observability tools allow for dynamic scaling adjustments based on real-time metrics, ensuring high availability and cost-efficiency without manual intervention or the rigid limitations imposed by fixed-size infrastructure configurations in production.

Manually deploy the model via the standard cluster user interface.

Why it fails: Manual deployment via cluster UI lacks reproducibility and auditability compared to using CI/CD pipelines and Model Serving APIs. Enterprise deployments should be automated through version-controlled scripts to ensure that every change is documented and tested, reducing human error and configuration drift in the production model serving environment.

The model is too large for the GPU memory.

Why it fails: A memory issue would manifest as an OOM (Out of Memory) error, not a network timeout related to a connection to an external vector store. This error clearly identifies a communication failure between the model service and the database, which is a network or configuration layer issue.

The model weights are corrupted.

Why it fails: Corrupted weights would cause errors during the model loading phase or during the inference computation itself, such as segmentation faults or arithmetic errors. It would not typically present as a connection timeout to an external dependency, as the model service would have already initialized before attempting a query.

The input prompt is too long for the model.

Why it fails: Prompt length limits result in model-specific errors, typically token limits or context window errors. They do not trigger network timeout errors regarding external service connections. This error is specific to the infrastructure communication path, not the content of the request being processed by the generative model.

Set the traffic_percentage of version 6 to 10 and keep version 5 at 100.

Why it fails: Traffic percentages must sum to 100%. Setting version 6 to 10 and version 5 to 100 results in a total of 110%, which is invalid for traffic routing configurations. The sum must always be exactly 100 to ensure predictable and correct distribution across the served model routes.

Remove version 5 and set version 6 to 10.

Why it fails: Removing version 5 while setting version 6 to 10 will result in 90% of traffic having nowhere to go, causing service errors. Canary deployments require that all traffic be accounted for across the defined routes. The sum of traffic percentages must always equal 100 for successful request routing.

Create a new endpoint entirely for version 6 and switch the DNS record.

Why it fails: Creating a new endpoint for a canary deployment is an unnecessarily complex and disruptive approach. Databricks Model Serving supports weighted traffic routing within a single endpoint. This feature is designed specifically to handle canary deployments and A/B testing natively without needing to manage external DNS records.

The endpoint will error because total traffic must equal 100% for each model version individually.

Why it fails: The configuration is valid as long as the total traffic distribution across all routes sums to 100%. In this case, 90% plus 10% equals 100%, which is a valid traffic policy. The system will correctly route requests according to these defined weights without errors.

The endpoint will only route traffic to churn_v1 after churn_v2 reaches capacity.

Why it fails: Traffic splitting in Databricks Model Serving is weight-based, not capacity-based. The system will continue to distribute requests according to the defined percentages regardless of the underlying load, provided the instances are healthy. It does not act as a failover mechanism based on capacity thresholds.

The endpoint will round-robin requests between v1 and v2 regardless of the weights.

Why it fails: Weights define the probability distribution for incoming requests. A round-robin approach would imply a 50/50 split, which contradicts the explicit 90/10 configuration provided in the exhibit. The routing engine respects the percentages defined in the traffic_config to ensure the desired distribution is maintained over time.

Delete the old model and immediately deploy the new version.

Why it fails: Deleting the old model before deploying the new one inevitably creates downtime, as the endpoint must be recreated. This is a risky, non-professional approach that is unsuitable for production systems, especially for generative AI applications that users rely on continuously for their daily business tasks and information needs.

Deploy all updates directly to the production endpoint.

Why it fails: Directly updating the production endpoint without a staged rollout or blue-green switch risks introducing breaking changes that immediately impact all users. This lack of a safety buffer makes it impossible to perform validation before deployment, leading to potentially poor outcomes if the new model version has issues.

Schedule deployments during peak traffic hours.

Why it fails: Deploying during peak hours is the worst possible strategy, as it maximizes the number of users affected by any potential deployment failures or latency spikes. Deployments should always occur during low-traffic windows and be supported by robust, zero-downtime strategies to protect the user experience and service reliability.

The model must be stored in a legacy Databricks Workspace folder.

Why it fails: Legacy workspace paths are not compatible with the modern Unity Catalog serving architecture. All models must be registered within Unity Catalog to leverage the unified governance model, lineage tracking, and permission structures that Mosaic AI services require to function effectively within a secure enterprise data environment.

The model must be served using a shared-access mode interactive cluster.

Why it fails: Model serving endpoints use dedicated, managed inference infrastructure, not interactive clusters. Interactive clusters are designed for development and exploratory analysis, whereas inference services require specific, optimized runtime environments that provide scalability, high availability, and auto-scaling capabilities tailored specifically for model serving workloads rather than general-purpose data science tasks.

The application code must perform manual model sharding across nodes.

Why it fails: Databricks Mosaic AI handles model sharding and distribution automatically within the managed serving environment. Developers do not need to manually manage hardware-level parallelism or model sharding, as the infrastructure layer abstracts these complexities to focus on ease of deployment, scalability, and simplified management of inference workloads.

Increase the number of retrieved chunks for every turn to compensate for conversational ambiguity.

Why it fails: Retrieving more chunks does not resolve ambiguity caused by unresolved references; it adds noise and token cost. A larger context can even distract the model from the most relevant passage. The better fix is query rewriting so the retriever receives a clear, standalone question.

Store the conversation history in the prompt and pass it unchanged to the retriever as the search query.

Why it fails: Passing the entire history as the search query dilutes the embedding with earlier topics and can retrieve irrelevant chunks. The retriever is optimized for a focused query, not a transcript. History should inform a rewritten query, not be used verbatim for similarity search.

Disable metadata filtering during multi-turn conversations so more historical documents are eligible.

Why it fails: Disabling metadata filters weakens governance and can surface content that should be excluded, without improving reference resolution. Multi-turn accuracy depends on query clarity, not on broadening the candidate set. Filters should remain in place to preserve policy and relevance.

The file format of the source documents (e.g., PDF vs. Word).

Why it fails: While file formats dictate how you extract text, they are not the primary factor for chunking. Once the text is extracted as a string, the chunking strategy is determined by the embedding model's requirements and the semantic structure of the content, not the original container file's metadata.

The total number of documents in the corpus.

Why it fails: The total number of documents influences scaling, but not the chunking strategy itself. Whether you have ten documents or ten million, the chunking strategy should be based on the model's capabilities and the nature of the information. Focusing on total count is a distraction from the technical requirements.

The storage cost of the resulting vector database.

Why it fails: While smaller chunks lead to more vectors, storage costs are rarely the primary driver of a chunking strategy. The primary goal is retrieval accuracy, not storage optimization. Prioritizing cost over accuracy leads to suboptimal RAG performance, making this an incorrect focus for data preparation strategy.

The user lacks the SELECT privilege on the schema object.

Why it fails: The SELECT privilege is not applicable to schema objects in Unity Catalog; permissions for schemas are governed by the USAGE privilege. The failure occurs because the user lacks USAGE on the schema, not because they are missing SELECT permissions on the schema container itself.

The user requires the MODIFY privilege to perform joins.

Why it fails: A join operation is a read-only activity that only requires the SELECT privilege on the underlying tables. The MODIFY privilege is required for DML operations like INSERT, UPDATE, or DELETE, and it is not a requirement for performing complex read operations or joining datasets.

The user needs to be an owner of the tables to perform joins.

Why it fails: Table ownership is not required for users to query tables. As long as the user has been granted the necessary SELECT and USAGE privileges via the Databricks SQL GRANT command, they can access and join tables regardless of who originally created or currently owns the table objects.

The catalog owner needs to refresh the table metadata.

Why it fails: Refreshing metadata does not propagate permissions to new objects. The issue is not related to metadata freshness but rather the fundamental way Unity Catalog handles object identity; each re-creation generates a distinct object that inherits no permissions from its predecessor.

The analyst group needs to be re-added to the schema permissions.

Why it fails: While schema permissions remain, table-specific permissions are lost. Re-adding the group to the schema would not restore the specific SELECT privilege on the new table object, because that privilege is tied to the specific table object that was deleted and replaced.

The user who recreated the table is not the catalog owner.

Why it fails: Ownership does not solve the underlying issue of permission binding. Even if the catalog owner recreated the table, the previous SELECT permissions for the analyst group would remain lost. Permissions are object-specific, not attached to the user who performed the DDL operation.

To increase compute capacity for the primary model.

Why it fails: Traffic splitting does not manage compute capacity. The number of instances and the resources allocated to each model version are determined by the scaling policy and workload types, not the traffic distribution settings, which simply control the routing of user requests between the available versions.

To load balance between two different cloud regions.

Why it fails: The 'served_model_name' refers to specific versions of a model deployed on the same serving endpoint. Traffic routing within a serving endpoint is local to the service and does not involve multi-region load balancing, which is a separate infrastructure-level configuration that is not managed through this endpoint routing policy.

To force all traffic to model-v1 when model-v2 errors.

Why it fails: This configuration represents a static split, not an automatic failover policy. If 'model-v2' returns errors, the endpoint will continue to route 10% of traffic to it. The system does not possess built-in intelligence to detect model-level errors and automatically redirect traffic back to the primary model.

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL) ON VIOLATION DROP ROW, and inspect the event log for dropped records.

Why it fails: ON VIOLATION DROP ROW discards violating records from the target table, which is exactly what the team wants to avoid. The event log records that rows were dropped, but the malformed transcripts are no longer available for review or reprocessing. Quarantine requires retaining the bad records in a separate location, not deleting them from the pipeline.

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL) ON VIOLATION DROP ROW, and configure the pipeline to write dropped rows to a Unity Catalog volume automatically.

Why it fails: Delta Live Tables does not automatically route dropped rows to a Unity Catalog volume; DROP ROW simply removes them from the target. The event log records counts and violation metrics but not the row payloads. This option promises a quarantine mechanism that the product does not provide, so malformed transcripts would be lost.

Define the table with a CONSTRAINT valid_role EXPECT (role IS NOT NULL) ON VIOLATION FAIL UPDATE, and rely on pipeline retries to reprocess the batch.

Why it fails: FAIL UPDATE aborts the entire update when a violation occurs, so a single malformed transcript stops the pipeline and blocks all valid records behind it. Retries will hit the same violation and fail again. This provides neither quarantine nor progress, and it is far more disruptive than the drop behavior the team wants to replace.

Chain-of-thought prompting

Why it fails: Chain-of-thought prompting asks the model to expose intermediate reasoning steps so that a complex task is solved more reliably. In this scenario, the engineer is not trying to elicit a reasoning trace from the answering model; they are inserting an independent LLM call whose only job is to judge and filter retrieved chunks. Chain-of-thought would not remove irrelevant chunks from the context window, so it does not solve the precision problem described.

Fine-tuning the embedding model on domain data

Why it fails: Fine-tuning an embedding model changes how vectors are produced and can improve recall for domain vocabulary. However, it does not add a separate evaluation step that inspects each retrieved chunk and discards those that fail a relevance check. The scenario explicitly describes a second LLM call that evaluates chunks, which is a runtime filtering pattern rather than a training-time embedding adjustment.

Hybrid search combining vector similarity with BM25 keyword matching

Why it fails: Hybrid search merges dense vector results with sparse keyword results to improve recall for exact terms. It still returns a ranked list of chunks and does not ask an LLM to judge whether each chunk contains enough information to answer the question. The scenario's second LLM call that filters chunks is the defining element, and hybrid search alone would not perform that filtering.

Partition the Delta table by the product category column so the embedding job reads fewer files.

Why it fails: Partitioning by category reduces scan scope only when queries filter on that column, but the embedding job still has no way to distinguish changed rows from unchanged ones within a partition. It cannot avoid reprocessing rows that were already embedded, so the core problem remains.

Convert the table to a view that filters rows by the current timestamp so only recent rows appear.

Why it fails: Filtering by timestamp assumes the table has a reliable modification timestamp column, which an upsert pipeline may not maintain, and it would miss updates to older rows. It also cannot distinguish which rows were already embedded, so it does not provide reliable incremental processing.

Run OPTIMIZE with Z-ORDER on the product identifier column to compact small files before each embedding run.

Why it fails: Compaction and Z-ORDER improve read performance by reducing file count and clustering related data, but they do not expose which rows changed. The embedding job would still read the full table each hour, so reprocessing is not eliminated.

Store the documents in DBFS and apply a cluster-level init script that filters results before returning them.

Why it fails: DBFS does not provide fine-grained access control comparable to Unity Catalog, and an init script runs at cluster startup, not per query. It cannot dynamically filter retrieval results based on the calling user's identity, so it fails to prevent unauthorized access to confidential documents.

Configure the endpoint with a service principal and set the vector search index to public read access.

Why it fails: Public read access on the index directly contradicts the requirement that only authorized users see retrieved content. A service principal authenticates the application, not the end user, so it cannot enforce per-user authorization for confidential financial documents.

Use a personal access token in the application code and rely on Unity Catalog table ACLs on the source documents.

Why it fails: Embedding a personal access token in application code is a security anti-pattern and does not enforce per-user access at query time. Unity Catalog ACLs on source tables do not automatically propagate to a Vector Search index, so retrieval could still return unauthorized content to the endpoint caller.

Move the clinical notes to an external location outside Unity Catalog and grant access via cloud IAM roles.

Why it fails: Moving data outside Unity Catalog removes the centralized governance and audit trail the auditor expects. Cloud IAM roles do not provide fine-grained column-level masking or Unity Catalog lineage. This approach also complicates RAG integration because the data would no longer be managed as a Unity Catalog table.

Grant the RAG application's service principal SELECT on the table and rely on the application to redact note text in its responses.

Why it fails: Relying on the application to redact text is not enforceable by Unity Catalog and cannot be proven to an auditor. The service principal would still have direct read access to raw note text, so any compromise or logging gap could expose PHI. This approach fails the requirement of demonstrable access control at the data layer.

Use a dynamic view that filters rows based on the user's group and grant the RAG service principal access to that view.

Why it fails: A dynamic view can filter rows, but it does not mask the note text column itself. If the RAG service principal needs embeddings, it would still require access to the underlying text to generate them, or the view must expose the text. This does not satisfy the requirement to prevent unauthorized viewing of raw note text.

groundedness

Why it fails: groundedness checks whether the answer is supported by the provided context, which is critical for RAG but not for distinguishing answer completeness. An answer that omits details can still be fully grounded in the context. Therefore, adding groundedness would not highlight the difference in supporting detail between the two templates.

toxicity

Why it fails: toxicity detects harmful or offensive language in the response. The scenario describes answers that are correct but incomplete, with no indication of inappropriate content. Adding toxicity would not reveal missing details and is irrelevant to the completeness problem the team is trying to diagnose.

answer_similarity

Why it fails: answer_similarity measures semantic similarity between the model output and the ground truth, often using embeddings. It can reward answers that are topically close even if they lack specific details, so it may not distinguish a complete answer from a vague but semantically similar one. It would not reliably surface the missing supporting details described in the scenario.

Read the Delta table that stores the evaluation results and compute a mean over the per-row correctness column to derive a run-level score.

Why it fails: The results table holds per-row assessments, not the aggregated run-level metrics, and averaging a per-row judge column can diverge from how the framework aggregates. While useful for error analysis, this recomputation is fragile and does not give the job a stable, framework-defined metric to compare against the baseline.

Re-run mlflow.evaluate() with the same evaluation dataset and inspect the printed summary in the notebook output to confirm the metrics match the baseline.

Why it fails: Re-running evaluation recreates judge calls and does not persist a comparison against the earlier baseline; the printed notebook summary is not machine-readable inside the job. It also costs additional judge invocations and can drift between runs, so it cannot serve as a reliable programmatic gate for the next deployment.

Enable inference tables on the serving endpoint and compare the latest logged request payloads to the baseline metric values stored in the model registry.

Why it fails: Inference tables capture live serving traffic, not offline evaluation runs, and the model registry stores model versions and tags rather than evaluation metric values. Comparing payloads to registry entries mixes two unrelated data sources, so the job would have no consistent numeric baseline to gate the deployment on.

Disable hybrid search and rely solely on dense vector similarity, since keyword matching introduces nondeterministic scoring.

Why it fails: Keyword matching is deterministic, so it cannot be the source of run-to-run variation, and removing it discards the exact-match signal that helps keyword-distinct queries. Disabling hybrid search would worsen the reported relevance problem rather than fix it, making this option contrary to the goal of improving retrieval quality for the described query patterns.

Set a fixed random seed on the embedding model so that repeated queries always produce identical similarity scores.

Why it fails: Embedding models are deterministic for a given input, so run-to-run variation is not caused by missing seeds. The observed inconsistency comes from how candidates are ranked and truncated, not from nondeterministic embeddings. Setting a seed therefore does not address the relevance or stability problem and misdiagnoses the root cause of the reported behavior.

Increase the number of results returned by the index and pass all of them, unchanged, directly into the LLM prompt as context.

Why it fails: Returning more candidates without reranking increases the chance that irrelevant chunks enter the prompt, which can degrade answer quality and waste context window. It also does not fix the ranking problem that causes keyword-distinct queries to miss relevant chunks, so result relevance and consistency are not meaningfully improved by this change alone.

Upgrade the model serving endpoint to a higher capacity cluster size.

Why it fails: Increasing cluster capacity only affects compute resource limits, not security or authorization policies. The error clearly indicates an authorization failure, meaning that compute resources are irrelevant until the identity used for deployment is granted the required permissions to access the specified model registry object.

Delete the model serving endpoint and recreate it using a personal access token.

Why it fails: Using a personal access token is discouraged for automated CI/CD deployments as tokens are linked to individual users and expire. Recreating the endpoint does not address the underlying permission issue, as the service principal will continue to lack access to the model regardless of how the endpoint is created.

Restart the Databricks workspace to force an update of the internal permission cache.

Why it fails: Permission errors in Unity Catalog are evaluated in real-time by the access control service. Restarting the workspace does not clear or refresh permissions. The issue is a missing grant, which requires a deliberate administrative action to update the security policy associated with the specific service principal account.