Design Applications practice questions

The endpoint will crash due to lack of static resource allocation.

Why it fails: Auto-scaling is specifically designed to prevent crashes by dynamically adding capacity when load increases. The system monitors utilization and adjusts the number of cores based on the defined range, ensuring that the service remains available even during periods of high demand without requiring permanent over-provisioning of resources.

The endpoint will force requests to queue indefinitely until traffic subsides.

Why it fails: The primary purpose of auto-scaling is to increase throughput capacity, not to throttle requests. By scaling up resources, the system accommodates increased demand in real-time, preventing the need for request queuing that would otherwise lead to poor user experiences and latency issues in production generative AI deployments.

The configuration is invalid because it lacks a GPU definition.

Why it fails: While GPUs are often used for LLM inference, CPU-based serving is a valid configuration for many models depending on size and performance requirements. The provided JSON is syntactically valid for a CPU-backed endpoint, and the absence of a GPU does not make the configuration inherently invalid or dysfunctional.

Configuring the endpoint to be publicly accessible from the internet.

Why it fails: Publicly accessible endpoints are a significant security risk for enterprise applications. Sensitive generative AI deployments should use private networking, such as Private Links, to ensure that traffic is kept within the internal network, protecting the data from unauthorized access and potential data exfiltration attempts over the public internet.

Allocating infinite CPU resources to prevent any latency.

Why it fails: Infinite resource allocation is not possible or practical. It would lead to massive, uncontrolled costs and could potentially starve other critical processes within the workspace. Scaling limits must be defined based on the expected load and budget, balancing performance requirements with operational cost and infrastructure governance.

Store all sensitive data in cleartext within the vector index for faster retrieval.

Why it fails: Storing sensitive data in cleartext within a vector index is a major security risk. Vector indexes are often stored in specialized structures where standard file-level permissions may not apply, making it impossible to audit access or enforce granular data masking policies effectively across your organization.

Perform embedding generation as a batch process at the time of retrieval.

Why it fails: Generating embeddings at retrieval time introduces massive latency, leading to a poor user experience. Embeddings should be pre-computed and stored in the vector index. Computing them on-the-fly forces the application to perform heavy ML tasks during the request-response cycle, which is highly inefficient and unscalable.

Expose the LLM endpoint directly to the public internet without authentication.

Why it fails: Exposing LLM endpoints without authentication allows unauthorized users to abuse resources, potentially leading to massive costs or data exfiltration. All enterprise applications must use robust authentication, such as Databricks Personal Access Tokens or Service Principals, to ensure that only authorized entities can interact with the models.

A manual spreadsheet to log every user interaction for offline review.

Why it fails: Manual logging via spreadsheets is unscalable and error-prone in production environments. It does not provide real-time monitoring, automated alerting, or integrated analytics, which are necessary for maintaining modern enterprise AI applications. It fails to meet the needs for observability and immediate response to performance issues.

A hardcoded threshold for total token usage per day.

Why it fails: While cost monitoring is useful, a simple hardcoded token threshold does not provide insight into performance, quality, or retrieval accuracy. It is a financial control mechanism rather than an application observability tool. It does not help in diagnosing why an application might be providing low-quality responses.

Databricks Repos

Why it fails: While Repos manages code versioning, it is not optimized for managing LLM prompts as model artifacts. Prompts need to be integrated into the model lifecycle, including tracking metadata, testing, and deployment, which MLflow handles natively. Repos is for file-level version control, not specific ML configuration objects.

Unity Catalog Volumes

Why it fails: Unity Catalog Volumes are designed for managing unstructured data files like images or PDFs. They do not provide the metadata management, versioning, or logging capabilities required for LLM prompts. Using volumes would treat prompts as generic files, ignoring the need for structured experimentation and model lifecycle management.

Delta Lake Constraints

Why it fails: Delta Lake constraints are used for data quality enforcement within tables, such as ensuring columns do not contain null values. They are unrelated to prompt management or the generative AI lifecycle. Attempting to use table constraints for prompts would be architecturally incorrect and ineffective for versioning.

Set the LLM temperature to a high value like 1.0 or greater.

Why it fails: High temperature increases randomness, which encourages the LLM to deviate from the provided context. For fact-based RAG applications, a low temperature (e.g., 0.0 to 0.2) is preferred to ensure the model focuses on the provided documentation rather than generating creative or potentially incorrect content.

Increase the number of retrieved chunks to include more irrelevant noise.

Why it fails: Retrieving irrelevant information introduces noise, which can actually increase the probability of hallucinations. The model may become confused by the contradictory data or focus on irrelevant parts of the context. High-quality RAG focuses on precision in retrieval to ensure the model is fed only relevant, verified information.

Write raw question and answer pairs to a Delta table using a custom callback handler registered in the chain

Why it fails: A custom callback can persist data, but it captures only what the callback is wired to observe and requires bespoke code for each chain stage, making intermediate retrieval results easy to miss. It also stores unstructured text without span hierarchy, so reconstructing a failed conversation's retrieval path becomes manual work rather than a trace view.

Log the final prompt string to the driver's stdout and rely on cluster log delivery for retention

Why it fails: Stdout logging captures only whatever the code explicitly prints, typically the final prompt, and loses the structure of retrieval steps and their timing. Cluster log delivery is also not designed as a queryable trace store, so correlating a failed conversation with specific retrieved documents becomes impractical and retention is limited.

Stream all chain events to an external observability SaaS platform using its LangChain integration

Why it fails: Sending chain events to an external SaaS platform transmits raw user question text outside the workspace boundary, which directly violates the stated compliance requirement. Even though the integration is technically straightforward, it fails the boundary constraint that only the approved serving endpoint may carry data outward.

Disable hybrid search and rely solely on dense vector similarity, since keyword matching introduces nondeterministic scoring.

Why it fails: Keyword matching is deterministic, so it cannot be the source of run-to-run variation, and removing it discards the exact-match signal that helps keyword-distinct queries. Disabling hybrid search would worsen the reported relevance problem rather than fix it, making this option contrary to the goal of improving retrieval quality for the described query patterns.

Set a fixed random seed on the embedding model so that repeated queries always produce identical similarity scores.

Why it fails: Embedding models are deterministic for a given input, so run-to-run variation is not caused by missing seeds. The observed inconsistency comes from how candidates are ranked and truncated, not from nondeterministic embeddings. Setting a seed therefore does not address the relevance or stability problem and misdiagnoses the root cause of the reported behavior.

Increase the number of results returned by the index and pass all of them, unchanged, directly into the LLM prompt as context.

Why it fails: Returning more candidates without reranking increases the chance that irrelevant chunks enter the prompt, which can degrade answer quality and waste context window. It also does not fix the ranking problem that causes keyword-distinct queries to miss relevant chunks, so result relevance and consistency are not meaningfully improved by this change alone.

Enable Foundation Model APIs pay-per-token for the chat model and keep the Vector Search index in a separate workspace.

Why it fails: Moving the chat model to pay-per-token Foundation Model APIs changes billing and capacity behavior, not the network path between the serving endpoint and the vector index. Keeping the index in a separate workspace adds cross-workspace authentication and network hops, which increases rather than reduces retrieval latency for a 40 GB knowledge base.

Configure the serving endpoint to use scale-to-zero so idle replicas are eliminated during low-traffic periods.

Why it fails: Scale-to-zero reduces cost during idle periods but introduces cold-start delays when requests resume, which directly conflicts with a one-second first-token requirement. It also does nothing to reduce the network round trips between the serving endpoint and the vector store, which is the actual source of latency in this scenario.

Increase the embedding dimensions of the Vector Search index from 1024 to 3072 to improve retrieval precision.

Why it fails: Increasing embedding dimensions improves semantic precision but also increases index size and per-query computation, which typically raises latency rather than lowering it. This does not address the network round-trip bottleneck described in the scenario, so it would worsen the first-token time while only marginally changing answer quality.

Store embeddings in a Unity Catalog volume and use a Python dictionary cached on the driver for lookups.

Why it fails: Loading millions of embeddings into a driver-side Python dictionary is not scalable, is not distributed, and will exhaust driver memory. Unity Catalog volumes are for file storage, not for serving vector similarity queries with low latency. This design also has no mechanism to stay synchronized with a Delta table receiving near-real-time updates, so the chatbot would serve stale answers.

Export the corpus to Parquet files and use a Spark UDF to compute cosine similarity at query time.

Why it fails: Computing cosine similarity with a Spark UDF at query time means scanning and scoring millions of vectors per request, which cannot meet sub-second latency. It also forces re-embedding whenever the model changes because there is no managed index, and it does not track Delta table updates incrementally. This approach is a batch analytics pattern, not a low-latency retrieval design.

Use MLflow Model Registry to version the embedding model and call the model serving endpoint directly for each document retrieval.

Why it fails: MLflow Model Registry versions models, but it does not provide a vector index or similarity search. Calling a model serving endpoint per document retrieval would require passing candidate documents and would not scale to 2 million documents with sub-second latency. This design conflates model lifecycle management with retrieval infrastructure and leaves synchronization with the Delta table unaddressed.

A single-node CPU cluster with the model loaded from a Unity Catalog volume at request time.

Why it fails: CPU-only inference for a Llama-class model is far too slow for interactive GenAI traffic and cannot scale horizontally to absorb a 200 requests-per-second burst. Loading the model from a Unity Catalog volume on each request adds cold-start latency and disk I/O, making the design unsuitable for bursty, low-latency serving.

Pay-per-token Foundation Model APIs with a fixed concurrency limit of two requests.

Why it fails: A fixed concurrency limit of two requests would throttle the application during spikes and create queueing that violates the bursty traffic requirement. Pay-per-token billing also does not provide the reserved GPU capacity needed to guarantee low latency for a fine-tuned custom model, which is not served by Foundation Model APIs anyway.

Provisioned throughput with a fixed number of concurrent requests sized to the 200 requests-per-second peak.

Why it fails: Provisioned throughput reserves capacity for a fixed workload and is billed for that reservation even when idle. Sizing it to the 200 requests-per-second peak defeats the goal of avoiding peak provisioning and wastes budget during the frequent low-traffic periods described. It is the right choice only when traffic is predictable and sustained, not bursty.

A custom Python script running on a single-node cluster that polls the volume for new files and updates the index.

Why it fails: A custom script on a single node is not scalable or reliable for large document collections. It lacks fault tolerance, monitoring, and automatic retries. Polling is inefficient and may miss files or cause delays. This approach does not leverage Databricks managed services and is not suitable for production-grade ingestion pipelines.

Databricks SQL with a scheduled query that reads new files and writes embeddings to a Delta table.

Why it fails: Databricks SQL is designed for SQL analytics, not for complex text extraction and embedding computation. It cannot easily call embedding models or manage Vector Search index updates. While it can schedule queries, it lacks the flexibility and ML capabilities needed for this pipeline. It is not the appropriate component for orchestrating document ingestion and indexing.

Databricks Jobs with a notebook that runs on a schedule to process new files and update the Vector Search index.

Why it fails: Scheduled notebooks can process new files, but they do not automatically detect new documents or handle incremental updates efficiently. They also lack built-in data quality checks and lineage. While Jobs can orchestrate tasks, a more automated and reliable approach is needed for continuous ingestion. This option requires manual scheduling and does not provide the seamless pipeline required for keeping the index current.

Call external LLM APIs from a standard Python notebook without VPC constraints.

Why it fails: Relying on external APIs introduces significant data egress risks and potential compliance violations for sensitive data. Without VPC constraints, data traffic is not contained within the secure workspace perimeter, making it unsuitable for organizations requiring strict internal governance and data residency control for their RAG pipeline components.

Export data to an external vector database and use a cloud-hosted LLM.

Why it fails: Exporting data outside the Databricks environment adds unnecessary complexity and increases the attack surface for sensitive information. Maintaining separate infrastructure for vector storage also complicates the governance model and lifecycle management of the RAG application, which should ideally be centralized within the Databricks ecosystem for optimal performance.

Use a public LLM endpoint with a public bucket to store the vector index.

Why it fails: Storing vector indexes in public buckets exposes them to unauthorized access, violating data security standards. Using public LLM endpoints also necessitates sending proprietary data over public networks, creating significant egress concerns and potential data leakage, which contradicts the fundamental requirements for a secure and private RAG application architecture.

The total number of parameters in the model regardless of the domain.

Why it fails: Model size is not a direct proxy for performance in niche domains. A massive general-purpose model may underperform compared to a smaller, fine-tuned model specifically trained on domain-specific vocabulary and technical documentation, leading to suboptimal retrieval accuracy for the intended use case while increasing unnecessary compute overhead for inference.

The availability of the model on the public Hugging Face repository.

Why it fails: While availability is convenient, the primary driver for selection should be performance and utility. Relying solely on availability ignores the necessity for models that can be deployed securely and efficiently within Databricks. Models must be evaluated for their functional capability rather than just their download source or public popularity.

The color scheme of the model's documentation page.

Why it fails: Visual aesthetics of documentation are irrelevant to the performance and utility of a machine learning model. Selection criteria should focus on technical capabilities, performance metrics, and infrastructure compatibility. Evaluating models based on irrelevant factors like documentation design ignores critical technical requirements needed for successful RAG application deployment and maintenance.

Increasing the chunk size to include the entire dataset in a single vector.

Why it fails: Storing entire datasets in a single vector dilutes the semantic information, making retrieval imprecise. Smaller, logically grouped chunks allow for more granular mapping to vector space, enabling the system to retrieve exactly the relevant context needed for a specific query, which is crucial for high-performance RAG systems.

Removing all stop words from the documents during the ingestion phase.

Why it fails: Removing stop words is generally an outdated practice for modern transformer-based embedding models. These models are designed to understand language syntax and context, and removing stop words can actually degrade the quality of the embeddings by stripping away linguistic nuances that the model uses to determine semantic meaning.

It automatically generates Python code for model training.

Why it fails: Unity Catalog is a governance and data management tool, not an automated code generation engine. Its purpose is to control access, track lineage, and manage metadata for data assets, not to replace the data engineering or machine learning code development process performed by the AI engineers in notebooks.

It converts unstructured text into vectors automatically.

Why it fails: While Unity Catalog manages the metadata of data, it does not perform data transformation or embedding tasks. The conversion of unstructured text into vectors is a separate process handled by embedding models within the ML workflow, not by the governance and storage management functions provided by Unity Catalog.

It eliminates the need for data preprocessing before RAG.

Why it fails: Data preprocessing, such as cleaning and chunking, is an essential step for RAG applications that cannot be bypassed by a governance tool. Unity Catalog ensures that the data is managed correctly, but it does not remove the technical requirements for preparing and structuring data for vector search.

Increasing the temperature parameter of the model to maximum.

Why it fails: Increasing the temperature makes the model more creative and random, which actually increases the likelihood of hallucinations. In RAG applications, a lower temperature is typically preferred to ensure that the model sticks strictly to the retrieved context, minimizing the risk of generating inaccurate or off-topic information.

Restricting the LLM to a specific list of keywords for its output.

Why it fails: Restricting output to keywords negates the natural language capabilities of an LLM. While it might prevent hallucinations, it results in an inflexible and unusable application. The goal of RAG is to leverage the generative power of the LLM while grounding it in relevant, up-to-date, and accurate source data.

Adding a long system prompt instructing the model not to lie.

Why it fails: While system prompts are important, they are insufficient to prevent hallucinations if the provided context is inherently stale or incorrect. Reliance on a prompt alone provides a false sense of security; the underlying data quality and the freshness of the retrieval index are far more critical for mitigating hallucination risk.

Unity Catalog audit logs.

Why it fails: Unity Catalog logs record access and governance events but do not evaluate the statistical performance or drift of ML models. While important for security, these logs do not provide the insights needed to monitor the quality or predictive accuracy of a generative model once it is deployed in production.

The Databricks SQL query history.

Why it fails: Query history tracks the performance of SQL queries against data warehouses but does not provide insight into model behavior or generative quality. While useful for database optimization, it lacks the specific capabilities required to assess the semantic accuracy or drift of an LLM or any other ML model.

Cluster event logs.

Why it fails: Cluster event logs provide information about infrastructure health and resource utilization, such as startup times or node failures. They are vital for system reliability but do not offer any information regarding the quality of the model’s outputs or its predictive behavior, which is the focus of model monitoring.

Disabling checkpointing to save storage costs.

Why it fails: Checkpointing is vital for long training jobs to allow for recovery in case of failures. Disabling it is poor practice because it risks losing hours of compute time if a node fails or a transient error occurs. Reliability in distributed training workflows is far more valuable than marginal storage savings.

Increasing the batch size to the maximum possible for all datasets.

Why it fails: While large batch sizes can improve training speed, they are constrained by available GPU memory. Attempting to force the maximum batch size often leads to out-of-memory errors. The batch size must be balanced with the available hardware resources to ensure stable and successful model training within the environment.

Embedding all user inputs into the vector database.

Why it fails: Embedding user inputs is for retrieval, not security. It does not provide any protection against malicious prompt injections. In fact, injecting user input into a vector store could be a vector for secondary attacks, as the system might retrieve and treat the malicious data as legitimate context.

Relying on the LLM’s internal safety training to reject injections.

Why it fails: While many modern LLMs have safety training, they are not impervious to sophisticated prompt injection attacks. Relying solely on the model's self-moderation is a security anti-pattern. An explicit, externalized security layer is necessary to ensure consistent and enforceable protection against adversarial inputs that could compromise the application.

Sending all inputs through a standard SQL query first.

Why it fails: SQL queries are for data retrieval and do not have the logic required to detect natural language injection attacks. Passing data through a SQL interface is irrelevant to the security of the LLM itself and does not protect against malicious prompts that are intended to manipulate the generative response.