Select Two (Multi-Select) Questions

Hardcode API credentials within the model inference script.

Why it fails: Hardcoding credentials is a severe security risk that exposes sensitive information to anyone with read access to the codebase. Instead, secrets management via Databricks Secrets or Unity Catalog should be used to securely inject credentials at runtime, maintaining a strict separation between code and sensitive configuration data.

Disable automatic scaling to maintain consistent latency.

Why it fails: Disabling automatic scaling does not improve observability and often leads to performance bottlenecks during traffic spikes. Proper observability tools allow for dynamic scaling adjustments based on real-time metrics, ensuring high availability and cost-efficiency without manual intervention or the rigid limitations imposed by fixed-size infrastructure configurations in production.

Manually deploy the model via the standard cluster user interface.

Why it fails: Manual deployment via cluster UI lacks reproducibility and auditability compared to using CI/CD pipelines and Model Serving APIs. Enterprise deployments should be automated through version-controlled scripts to ensure that every change is documented and tested, reducing human error and configuration drift in the production model serving environment.

A manual provisioning of a dedicated GPU worker pool

Why it fails: Databricks Model Serving is a serverless offering, meaning users do not need to manually provision or manage infrastructure like worker pools. The platform handles autoscaling, capacity management, and infrastructure allocation automatically based on the traffic demands, simplifying the deployment process for the developer.

A hardcoded API key for external vector store access

Why it fails: Hardcoding credentials is a significant security risk and is discouraged. Instead, Databricks provides secret scopes or Unity Catalog connections to handle authentication securely. Storing secrets within the code exposes the application to potential compromise, violating security best practices for production enterprise application deployments.

A pre-configured Kubernetes manifest file

Why it fails: Databricks Model Serving is a fully managed service that abstracts away Kubernetes complexity. Developers do not interact with K8s manifests or cluster configuration files. The platform handles containerization and orchestration internally, allowing developers to focus strictly on the agent logic rather than the underlying infrastructure.

Inference Latency

Why it fails: Inference latency is a performance metric, not a quality metric. While it is important for the operational health of the application, it does not inform you about the factual accuracy, relevance, or grammatical correctness of the content produced by the generative model during the interaction.

GPU Utilization

Why it fails: GPU utilization is a hardware-level monitoring metric used for infrastructure scaling and cost optimization. It provides zero insight into the quality of the generative AI outputs. Focusing on utilization while ignoring output quality is a common pitfall that overlooks the core objective of the AI application's success.

Training Iterations

Why it fails: Training iterations are a parameter of the fine-tuning process. They reflect the effort spent during development but tell you nothing about the quality of the resulting model's outputs in a production RAG environment. Monitoring must be based on inference-time results, not internal training process statistics.

The model must be stored in a legacy Databricks Workspace folder.

Why it fails: Legacy workspace paths are not compatible with the modern Unity Catalog serving architecture. All models must be registered within Unity Catalog to leverage the unified governance model, lineage tracking, and permission structures that Mosaic AI services require to function effectively within a secure enterprise data environment.

The model must be served using a shared-access mode interactive cluster.

Why it fails: Model serving endpoints use dedicated, managed inference infrastructure, not interactive clusters. Interactive clusters are designed for development and exploratory analysis, whereas inference services require specific, optimized runtime environments that provide scalability, high availability, and auto-scaling capabilities tailored specifically for model serving workloads rather than general-purpose data science tasks.

The application code must perform manual model sharding across nodes.

Why it fails: Databricks Mosaic AI handles model sharding and distribution automatically within the managed serving environment. Developers do not need to manually manage hardware-level parallelism or model sharding, as the infrastructure layer abstracts these complexities to focus on ease of deployment, scalability, and simplified management of inference workloads.

Increase the number of retrieved chunks for every turn to compensate for conversational ambiguity.

Why it fails: Retrieving more chunks does not resolve ambiguity caused by unresolved references; it adds noise and token cost. A larger context can even distract the model from the most relevant passage. The better fix is query rewriting so the retriever receives a clear, standalone question.

Store the conversation history in the prompt and pass it unchanged to the retriever as the search query.

Why it fails: Passing the entire history as the search query dilutes the embedding with earlier topics and can retrieve irrelevant chunks. The retriever is optimized for a focused query, not a transcript. History should inform a rewritten query, not be used verbatim for similarity search.

Disable metadata filtering during multi-turn conversations so more historical documents are eligible.

Why it fails: Disabling metadata filters weakens governance and can surface content that should be excluded, without improving reference resolution. Multi-turn accuracy depends on query clarity, not on broadening the candidate set. Filters should remain in place to preserve policy and relevance.

The file format of the source documents (e.g., PDF vs. Word).

Why it fails: While file formats dictate how you extract text, they are not the primary factor for chunking. Once the text is extracted as a string, the chunking strategy is determined by the embedding model's requirements and the semantic structure of the content, not the original container file's metadata.

The total number of documents in the corpus.

Why it fails: The total number of documents influences scaling, but not the chunking strategy itself. Whether you have ten documents or ten million, the chunking strategy should be based on the model's capabilities and the nature of the information. Focusing on total count is a distraction from the technical requirements.

The storage cost of the resulting vector database.

Why it fails: While smaller chunks lead to more vectors, storage costs are rarely the primary driver of a chunking strategy. The primary goal is retrieval accuracy, not storage optimization. Prioritizing cost over accuracy leads to suboptimal RAG performance, making this an incorrect focus for data preparation strategy.

Provides a graphical drag-and-drop interface for deployment

Why it fails: DABs are a CLI-based tool, not a graphical interface. The shift away from drag-and-drop UI configurations is a core benefit, as it eliminates the 'click-ops' anti-pattern. This ensures that deployments are documented, reproducible, and less prone to the human errors associated with manual UI-based configuration.

Automatically handles data labeling and cleaning tasks

Why it fails: DABs manage the deployment of infrastructure resources like jobs and models; they do not perform data processing, labeling, or cleaning tasks. Data preparation is an application-level responsibility that must be implemented within the code or pipelines that the bundle deploys, not a function of the bundle itself.

Allows users to bypass Unity Catalog governance

Why it fails: DABs do not bypass governance; in fact, they fully support and encourage the use of Unity Catalog to manage assets. Any attempt to use DABs to circumvent security or access controls would be blocked by the underlying Databricks platform's security architecture, which enforces Unity Catalog permissions.

Embed all logic including data preprocessing in a single, large notebook.

Why it fails: Large, monolithic notebooks are difficult to debug, test, and maintain. When inference logic is mixed with preprocessing, versioning and testing become complex and error-prone. Modularizing the code into separate notebooks or library functions is a standard best practice that significantly improves code quality and reduces the risk of deployment errors.

Hardcode all file paths and configuration settings for consistency.

Why it fails: Hardcoding paths and settings is a poor practice that limits the notebook's portability. If the model is moved to a new environment or the data location changes, the code must be manually edited. This increases the likelihood of human error and makes the pipeline rigid, difficult to scale, and hard to update.

Always run inference on the driver node to avoid network latency.

Why it fails: Running inference on the driver node is a bottleneck that prevents parallelization and fails for large datasets. Databricks is designed for distributed processing; inference should be distributed across worker nodes using Spark to maximize performance. Relying on the driver node for compute-intensive tasks is an anti-pattern that leads to poor scalability.

Deploy the candidate template to a small percentage of production traffic and rely on user thumbs feedback alone as the promotion signal.

Why it fails: Online canary feedback is valuable but sparse, slow, and self-selected, so a small traffic slice rarely accumulates enough signal to detect a modest regression before the decision deadline. Offline paired evaluation on a curated dataset provides denser, faster, and more controlled evidence; the canary is a complement, not the primary gate.

Increase the judge model size until its scores on the candidate template exceed the production template's scores.

Why it fails: Scaling the judge upward until a desired outcome appears is tuning the measurement to the conclusion. Judge capability affects score calibration, not whether the candidate genuinely regressed. This practice invalidates the comparison because the two templates would no longer be measured under identical conditions, and it invites threshold gaming rather than evidence.

Evaluate only the questions where the candidate template produced different answers, skipping the rest to save judge tokens.

Why it fails: Skipping unchanged answers biases the sample toward cases where behavior moved, which may be exactly the regressions or exactly the improvements. It also removes the baseline needed to compute a per-question delta and can hide that an unchanged answer became incorrect due to a retrieval shift. Cost savings do not justify an unrepresentative evaluation set.

Disable the serving endpoint's logging to save on compute cycles.

Why it fails: Disabling logging does not meaningfully impact the inference performance of the serving endpoint, as logging is an asynchronous process. Furthermore, turning off observability prevents developers from diagnosing bottlenecks during peak load, which is counterproductive when trying to maintain and optimize the performance of the model serving infrastructure.

Configure the endpoint to run on a single, fixed-size node to minimize latency.

Why it fails: A single, fixed-size node is prone to becoming a bottleneck during peak traffic, leading to increased latency and potential service outages. Scaling horizontally across multiple nodes is the best practice for high availability, allowing the workload to be distributed effectively and ensuring resilience against high traffic demands.

Manually restart the endpoint every hour to clear the cache.

Why it fails: Restarting an endpoint causes unnecessary downtime and flushes the model cache, which can actually increase latency for subsequent requests as the model reloads. This is an anti-pattern that disrupts availability and does not address the underlying need for scalable infrastructure, which should be managed by the platform automatically.

Store the full PDF text in a single chunk per document

Why it fails: One chunk per document maximizes topic mixing and dilutes the embedding across the entire document, making similarity search less discriminating. Large chunks also consume prompt budget and bury the relevant passage among irrelevant text, worsening the exact symptoms the engineer is trying to fix.

Increase the embedding dimension of the model used for the index

Why it fails: Embedding dimension affects representational capacity, not the internal coherence of a chunk. Multi-topic chunks would still retrieve poorly because their vectors average several themes, so a higher-dimensional model does not fix the underlying chunking problem and adds storage and query cost.

Lower the temperature setting on the language model

Why it fails: Temperature influences sampling randomness in generation, not what context is retrieved or how well a chunk supports a claim. Vague answers driven by poor chunk boundaries persist regardless of temperature, so this change does not address the ingestion-stage defect.

Unity Catalog volumes to store the model weights and configuration files.

Why it fails: Volumes are used for storing and accessing files, but they do not provide fine-grained access control or auditing for model invocations or vector search queries. While you can store model artifacts in volumes, they do not log inference requests or enforce privileges on index queries. Thus, volumes alone do not meet the governance requirements.

Cluster policies to restrict the types of clusters that can access the data.

Why it fails: Cluster policies control cluster configurations and who can create clusters, but they do not govern data access or model invocations. They cannot enforce fine-grained privileges on Unity Catalog objects or log inference requests. Thus, cluster policies are not suitable for meeting the stated governance requirements.

Databricks SQL dashboards to visualize access patterns and model usage.

Why it fails: Dashboards are visualization tools and do not enforce access control or provide raw audit logs. They can display aggregated metrics, but they cannot capture per-request details or restrict who can query the index or invoke the model. Therefore, dashboards are not a governance mechanism for fine-grained access and auditing.

Disabling checkpointing to save storage costs.

Why it fails: Checkpointing is vital for long training jobs to allow for recovery in case of failures. Disabling it is poor practice because it risks losing hours of compute time if a node fails or a transient error occurs. Reliability in distributed training workflows is far more valuable than marginal storage savings.

Increasing the batch size to the maximum possible for all datasets.

Why it fails: While large batch sizes can improve training speed, they are constrained by available GPU memory. Attempting to force the maximum batch size often leads to out-of-memory errors. The batch size must be balanced with the available hardware resources to ensure stable and successful model training within the environment.

The total number of parameters in the model regardless of the domain.

Why it fails: Model size is not a direct proxy for performance in niche domains. A massive general-purpose model may underperform compared to a smaller, fine-tuned model specifically trained on domain-specific vocabulary and technical documentation, leading to suboptimal retrieval accuracy for the intended use case while increasing unnecessary compute overhead for inference.

The availability of the model on the public Hugging Face repository.

Why it fails: While availability is convenient, the primary driver for selection should be performance and utility. Relying solely on availability ignores the necessity for models that can be deployed securely and efficiently within Databricks. Models must be evaluated for their functional capability rather than just their download source or public popularity.

The color scheme of the model's documentation page.

Why it fails: Visual aesthetics of documentation are irrelevant to the performance and utility of a machine learning model. Selection criteria should focus on technical capabilities, performance metrics, and infrastructure compatibility. Evaluating models based on irrelevant factors like documentation design ignores critical technical requirements needed for successful RAG application deployment and maintenance.

Increasing the chunk size to include the entire dataset in a single vector.

Why it fails: Storing entire datasets in a single vector dilutes the semantic information, making retrieval imprecise. Smaller, logically grouped chunks allow for more granular mapping to vector space, enabling the system to retrieve exactly the relevant context needed for a specific query, which is crucial for high-performance RAG systems.

Removing all stop words from the documents during the ingestion phase.

Why it fails: Removing stop words is generally an outdated practice for modern transformer-based embedding models. These models are designed to understand language syntax and context, and removing stop words can actually degrade the quality of the embeddings by stripping away linguistic nuances that the model uses to determine semantic meaning.

CPU utilization percentage.

Why it fails: CPU utilization is a system performance metric, not a quality metric for generative models. While important for infrastructure monitoring and cost management, it provides zero insight into whether the model is producing accurate, relevant, or hallucination-free content, making it irrelevant for assessing the quality of the generative application.

Network latency in milliseconds.

Why it fails: Network latency measures the responsiveness of the infrastructure, not the quality of the model's output. While essential for user experience, it does not assess whether the model is actually performing its task of providing accurate and relevant information, and it should be treated as a separate operational concern.

Total storage cost of the index.

Why it fails: Storage cost is a financial metric related to resource management and index size. It does not provide any information regarding the accuracy or semantic quality of the responses generated by the RAG system, making it completely unsuitable for evaluating the effectiveness of the model's reasoning or the retrieval accuracy.

Hardware temperature monitoring for GPUs

Why it fails: While GPU temperature is important for data center operations, it is an infrastructure metric that does not provide any insight into the effectiveness of the RAG pipeline or the quality of the generative AI outputs.

Network packet loss percentage

Why it fails: Network packet loss percentage measures transport reliability, not retrieval or generation quality, so it cannot assess grounding, relevance or faithfulness in a RAG pipeline. It is tempting because packet loss genuinely degrades distributed inference throughput, making it a valid metric for infrastructure capacity planning rather than application evaluation.

Enable public access on all input data buckets for faster retrieval.

Why it fails: Enabling public access on data buckets exposes the entire dataset to external risks, violating basic security principles and compliance regulations. Even if the data is intended for training, it must be protected via strictly defined access controls to ensure that only authorized services and users can interact with it.

Store all API keys as plain text in notebook variables.

Why it fails: Hardcoding credentials or storing them in plain text notebooks exposes sensitive tokens to any user with read access to the code. This is a severe security vulnerability that could lead to unauthorized API usage, unexpected billing spikes, or potential compromise of the entire application environment and infrastructure.

Use the model provider's default logging for all user queries.

Why it fails: Default logging often captures raw user inputs, which may include PII or sensitive corporate data. Relying on provider-side logging without thorough vetting or masking risks exposing proprietary information to the model vendor, which is typically unacceptable in enterprise environments that require full control over data handling and security.

An external third-party Hadoop cluster managed via SSH tunnels

Why it fails: External Hadoop clusters managed via SSH tunnels introduce severe operational overhead, security risks, and network latency. Modern Databricks applications leverage native serverless and cloud-optimized services rather than legacy distributed computing architectures for real-time inference tasks.

A manual cron job running on a local developer laptop

Why it fails: A cron job on a developer laptop provides no scalable, always-available retrieval path and cannot serve Databricks Vector Search indexes at low latency. It tempts for scheduled batch refresh of embeddings, but that scenario is offline index maintenance, not the online similarity-search architecture the application requires.

A static CSV file stored on local driver disk storage

Why it fails: Static CSV files stored on local driver storage cannot scale concurrently and lack transactional consistency, indexing capabilities, and high availability. Production RAG applications require distributed, ACID-compliant storage layers like Delta Lake alongside dedicated vector indexes.

Deploying the 'latest' alias for all production endpoints.

Why it fails: Using 'latest' aliases in production is dangerous because it can lead to unexpected model updates without sufficient validation. Production endpoints should always be pinned to a specific model version or a stable production alias to ensure consistency and allow for proper rollback if a new model version fails.

Using static credentials embedded directly in the deployment YAML.

Why it fails: Embedding credentials in configuration files is a major security vulnerability. It exposes sensitive information to anyone with access to the source code repository. Instead, secrets should be retrieved at runtime using the Databricks Secrets API or managed via environment-specific service principal configurations with limited scopes.