Application Development practice questions

Always set the warehouse size to '4XL' to ensure maximum compute power.

Why it fails: Over-provisioning compute for small, frequent requests is inefficient and costly. Large clusters are optimized for massive parallel processing of complex analytical queries, not for low-latency small requests. Excessive sizing introduces unnecessary overhead and increases costs without providing performance benefits for simple, single-row or small-batch result sets.

Disable the query result cache to ensure real-time data retrieval.

Why it fails: Disabling the query result cache forces the warehouse to re-compute results on every request, significantly increasing latency and cost. For applications requiring real-time data, proper cache invalidation or shorter TTLs are preferred over disabling the cache entirely, as the cache is a critical component for high-performance applications.

Store all application data in JSON files to optimize read performance.

Why it fails: JSON is a semi-structured format that is generally slower to query than highly optimized formats like Delta Lake. Delta Lake provides features such as data skipping, z-ordering, and compact file sizes, which are essential for achieving high performance in SQL Warehouses compared to reading raw, unindexed JSON files.

Store all vectors in a compressed Parquet file on S3 for batch retrieval.

Why it fails: Storing vectors in standard Parquet files is not optimized for low-latency similarity search. Querying flat files requires full scans unless complex, manual indexing is implemented, which would result in high latency. Parquet is excellent for bulk storage but unsuitable for the rapid nearest-neighbor lookups required by GenAI applications.

Provision dedicated, high-memory clusters solely for the embedding model.

Why it fails: Provisioning dedicated, high-memory clusters is expensive and doesn't directly solve retrieval latency; it only addresses the throughput of the embedding model itself. Latency is more heavily impacted by the storage-to-compute transfer and the search algorithm efficiency, which are better addressed by optimized vector search indexes than just hardware.

Increase the vector dimension size to ensure higher search accuracy.

Why it fails: Increasing the vector dimension size significantly increases the computational complexity of the similarity search, which directly leads to higher latency. While accuracy might improve, it does so at the cost of performance. For low-latency applications, developers often need to balance dimensionality with search speed constraints.

Assign the job to the 'admin' user to avoid permission errors.

Why it fails: Using the 'admin' account violates the security principle of least privilege. Production jobs should run with the minimum permissions required to complete their tasks. Relying on admin privileges increases the security surface area and makes auditing difficult, as it masks the actual permissions needed by the application.

Store the job execution logs in a public cloud bucket.

Why it fails: Storing logs in a public location introduces security risks and potential data leakage. Production logs often contain sensitive operational information or data samples that must be protected. Logs should be stored in secure, private locations with strict access controls to ensure compliance and internal security protocols.

Directly update the existing endpoint configuration with the new model version.

Why it fails: Direct updates trigger an immediate restart of the underlying infrastructure, causing significant downtime for active requests. This approach lacks a rollback mechanism, making it unsuitable for high-availability requirements where continuous access to the model inference service is mandatory during the deployment phase of the development lifecycle.

Pause the model serving endpoint before deploying the new version.

Why it fails: Pausing an endpoint effectively kills all active connections and rejects new requests until the process completes. This is antithetical to high-availability design patterns, as it forces an artificial outage that negatively impacts the end-user experience and breaks real-time integration workflows within the production environment.

Scale the model serving endpoint to zero and redeploy.

Why it fails: Scaling to zero results in a cold-start latency penalty when the endpoint is triggered again, and it terminates existing sessions. This strategy is primarily intended for cost optimization during inactivity rather than for managing production deployments where consistent, low-latency inference performance is a core requirement for applications.

Set the MLflow registry URI to the workspace-local path.

Why it fails: Workspace-local registries are legacy components and do not support Unity Catalog features. Using this URI prevents the model from being discoverable within the catalog, effectively siloing the artifact and making it impossible to apply unified access control policies or share the model across different workspaces.

Convert the model to a generic pickle format before logging.

Why it fails: Logging models as generic pickle files bypasses the automated signature inference and dependency management provided by MLflow flavors. This creates significant risks regarding environment reproducibility, as the system cannot guarantee the specific versions of libraries required to load the model during inference in a production environment.

Manually upload the model weights to the DBFS root.

Why it fails: Manual uploads to DBFS bypass the MLflow tracking system entirely, leading to a loss of metadata, training parameters, and lineage information. Without these artifacts, the model lacks the necessary documentation for audits and cannot be served through Mosaic AI Model Serving, rendering it effectively unusable for professional deployment.

Increase the number of dimensions in the vector embeddings.

Why it fails: Higher dimensions increase the computational complexity of the search process, leading to higher latency and increased memory usage. While high dimensions might offer better representational capacity, they are often counterproductive for latency-sensitive applications unless strictly necessary for capturing complex semantic nuances that smaller vectors cannot represent.

Perform exhaustive exact search to ensure 100% precision.

Why it fails: Exact search requires calculating the distance between the query and every document in the index. This approach does not scale as the dataset grows and is prohibitively slow for real-time RAG applications, making it unsuitable for production environments where user experience depends on rapid query turnaround times.

Cache all user queries in a global memory store.

Why it fails: While caching improves performance for duplicate queries, it does not improve the retrieval performance for new or varied queries, which are common in RAG applications. Caching provides a limited benefit and introduces complexity regarding cache invalidation when the underlying vector index data changes, making it an incomplete optimization strategy.

Hardcode the production endpoint name directly in the Python application script and use environment branching logic.

Why it fails: Hardcoding environment-specific names inside application code violates the fundamental design principles of Databricks Asset Bundles. It requires maintaining multiple code branches or complex conditional blocks, increasing maintenance overhead and the risk of deploying incorrect configurations to production environments.

Create three entirely separate repository checkouts, each containing manually edited configuration files for a single environment.

Why it fails: Maintaining separate repository checkouts for each environment violates DRY principles and drastically increases administrative overhead. Manual file editing invites human error, inconsistencies across environments, and makes auditing configuration drift extremely difficult as the software development lifecycle scales.

Store the endpoint name in a temporary SQLite database on the driver node during job execution initialization.

Why it fails: Using a temporary SQLite database on the driver node introduces unnecessary complexity and fails to integrate with Databricks deployment tooling. Driver nodes are ephemeral in serverless and automated job contexts, meaning configuration state stored locally will be lost across restarts and scaling events.

Provision the maximum possible GPU instance type permanently without enabling autoscaling to eliminate scaling latency.

Why it fails: Provisioning maximum compute permanently without autoscaling results in exorbitant idle costs during low-traffic periods. This violates cost management best practices and wastes expensive GPU resources when application demand is minimal.

Disable all rate limiting and concurrent request constraints to allow unlimited traffic processing on a single node.

Why it fails: Disabling concurrency limits and rate controls on a single node will quickly overwhelm the serverless instance, resulting in request queuing timeouts, out-of-memory errors, and complete application failure under high-volume production traffic.

Rely on global environment variables configured on the cluster driver node to inject libraries dynamically at inference time.

Why it fails: Relying on global cluster environment variables is an anti-pattern that breaks reproducibility. Model serving containers and scoring jobs run in isolated environments where global driver variables are not automatically available or injected into the worker runtime.

Manually copy the site-packages folder from the development laptop into the root of the MLflow artifact directory.

Why it fails: Manually copying site-packages introduces platform-specific binaries and compiled C extensions that are incompatible with Databricks container Linux environments. This method causes runtime crashes and violates standard MLflow artifact packaging protocols.

Databricks Delta Sharing

Why it fails: Delta Sharing is intended for securely sharing data with third parties across organizational boundaries. While it is a useful feature, it does not provide the internal lineage tracking or model governance capabilities required to audit the lifecycle of an AI model from training to deployment within an internal environment.

Databricks SQL Dashboards

Why it fails: Dashboards are for visualization and reporting on data insights, not for managing AI model lifecycles or tracking lineage. They cannot provide the deep, granular tracing of data flow from source tables through training pipelines to deployed endpoints, which is a requirement for modern AI governance and model development.

Databricks Repos

Why it fails: Repos provide Git integration for versioning code, but they do not manage the data or model lineage. While code versioning is a part of the development cycle, it is not the same as data and model lineage, which tracks how the actual data impacts the model behavior during the training phase.

Delta Lake

Why it fails: Delta Lake is a storage layer focused on data reliability and performance. While it is used to store the data that trains models, it is not a model registry. It lacks the versioning, staging, and lifecycle management capabilities required to manage machine learning model artifacts effectively in a workspace.

MLflow Tracking Server

Why it fails: The MLflow Tracking Server is designed for logging parameters, metrics, and artifacts during model training. While it is part of the MLflow ecosystem, it does not provide the centralized model governance, lifecycle staging, and cross-workspace permissions management that Unity Catalog provides for organizational model management and deployment tracking.

Databricks Workflows

Why it fails: Databricks Workflows is an orchestration service for scheduling jobs and managing pipelines. While it can trigger deployments, it is not a storage or governance layer for model artifacts. It lacks the metadata registry features needed to manage model versions and staging transitions in a centralized, secure repository.

Enable Unity Catalog on the workspace.

Why it fails: While Unity Catalog is highly recommended for governance, it is not a strictly mandatory architectural component for creating a Vector Search index. One can technically operate Vector Search without it, although doing so would limit the ability to share and govern the underlying data assets across the organization.

Install an external vector database like Pinecone or Milvus.

Why it fails: Vector Search is a native Databricks service. Installing external vector databases is unnecessary and counterproductive, as it adds complexity and latency. The platform's built-in capability is optimized for integration with Delta Lake and Databricks Model Serving, providing a more cohesive and performant experience for the AI developer.

Databricks SQL API

Why it fails: The Databricks SQL API is designed for executing SQL queries and managing the data warehouse environment. It is not capable of interacting with or managing Model Serving endpoints. Attempting to use this API for model lifecycle tasks will fail, as it lacks the necessary endpoints for inference-related infrastructure management.

Unity Catalog API

Why it fails: The Unity Catalog API is focused on governance, data lineage, and object management within the catalog. While it can retrieve information about model versions, it does not provide the capability to create, update, or manage the operational infrastructure of serving endpoints themselves, which is handled by the Model Serving API.

Jobs API

Why it fails: The Jobs API is designed to manage the orchestration of data pipelines and notebook executions. While it can be used to trigger the script that *calls* the Model Serving API, it is not the API responsible for managing the actual infrastructure of the serving endpoint itself.

Create a separate keyword index using Apache Lucene and query both indexes, merging results with reciprocal rank fusion.

Why it fails: While a separate keyword index can improve lexical matching, it does not address synonym gaps because keyword indexes rely on exact term matches. Merging results adds complexity and latency without solving the core synonym problem. Databricks Vector Search already provides integrated hybrid search if needed.

Increase the number of nearest neighbors returned by the Vector Search query to 100 and filter results by a minimum similarity score.

Why it fails: Increasing the number of neighbors can improve recall by returning more candidates, but it does not help if the embedding model fails to capture synonyms. If synonyms are not represented similarly in vector space, no amount of neighbor expansion will retrieve relevant documents. This approach also increases latency and may introduce noise.

Enable hybrid search on the Vector Search index, which combines dense vector similarity with keyword-based BM25 scoring.

Why it fails: Hybrid search combines semantic and keyword matching, but it does not inherently handle synonyms. It improves recall for exact keyword matches, not for terms that are semantically similar but lexically different. The scenario requires handling synonyms, which dense embeddings already address, but hybrid search alone does not guarantee synonym coverage.

mlflow.transformers

Why it fails: The transformers flavor is designed for Hugging Face models and pipelines. It does not natively support custom retrieval logic that combines a Vector Search index and a reranker. While you could wrap the retrieval in a transformers pipeline, it would be convoluted and not leverage the full capabilities of MLflow for custom Python code. The pyfunc flavor is more appropriate for this composite component.

mlflow.langchain

Why it fails: The langchain flavor is for LangChain models, but it is less flexible than pyfunc for custom retrieval components that may not be built entirely with LangChain. If the retrieval logic uses custom code or direct API calls to Vector Search, pyfunc is more appropriate. The langchain flavor also has specific requirements for model signatures and may not support all deployment scenarios on Mosaic AI Model Serving.

mlflow.sklearn

Why it fails: The sklearn flavor is for scikit-learn models. It is not suitable for packaging a retrieval component that involves Databricks Vector Search and a reranker, which are not scikit-learn estimators. Using this flavor would require significant workarounds and would not naturally support the required inference logic or dependencies.

Use MLflow's built-in langchain flavor to log a chain that includes Vector Search retrieval and LLM calls, then deploy to a serving endpoint.

Why it fails: MLflow does have a langchain flavor, but it may not support all Databricks-specific integrations like the Vector Search client seamlessly. Moreover, the langchain flavor can have dependency and serialization challenges. A custom PyFunc model offers more control and reliability for production RAG pipelines on Databricks, ensuring compatibility with Mosaic AI Model Serving.

Create a Databricks Job that runs a Python script to retrieve documents and call the LLM, and expose it via a REST API using Flask.

Why it fails: While a Databricks Job can run retrieval and generation, it is not designed for low-latency, real-time serving. Exposing it via a custom Flask API would require managing infrastructure, scaling, and authentication, which Mosaic AI Model Serving handles automatically. This approach is operationally complex and not integrated with MLflow model deployment.

Log the LLM using mlflow.transformers and deploy it directly, relying on the model to retrieve documents internally.

Why it fails: Logging a transformers model directly deploys only the LLM, which does not have built-in retrieval capabilities. The model would not know how to query Vector Search or incorporate external context. This approach fails to implement RAG; it would just generate responses based on the LLM's pretrained knowledge, not the specific documents.

Reduce the embedding dimension by using a smaller model, such as a 384-dimensional model instead of a 768-dimensional one.

Why it fails: Reducing embedding dimension can lower latency and storage, but it may also degrade retrieval quality because smaller models may capture less semantic nuance. The scenario requires maintaining retrieval quality, so this trade-off is risky. Moreover, the latency issue might be due to index configuration rather than dimension alone.

Enable the Vector Search index's automatic sync and increase the sync frequency to every minute.

Why it fails: Automatic sync frequency affects index freshness, not query latency. Increasing sync frequency may add background load but does not speed up individual queries. The latency issue is likely due to index size or query complexity, not sync interval. This change would not reduce the 2-second query time.

Increase the number of nearest neighbors returned by the Vector Search query from 5 to 20.

Why it fails: Increasing the number of nearest neighbors returns more candidates, which can improve recall but also increases the amount of data transferred and processed, potentially raising latency. It does not address the root cause of slow Vector Search queries. In fact, it may worsen latency because more vectors need to be ranked and returned.

Microsoft Entra ID token

Why it fails: Microsoft Entra ID tokens can be used for authentication in Azure Databricks, but the question specifies Databricks Mosaic AI Model Serving, which is a cross-cloud feature. The most portable and recommended method is OAuth from a Databricks service principal, which works across clouds. Microsoft Entra ID tokens may not be applicable in AWS or GCP deployments and require additional configuration.

Personal access token (PAT) stored in environment variables

Why it fails: While PATs can be used, storing them in environment variables is less secure than using Databricks secrets and can lead to accidental exposure. PATs also have broad permissions and lack fine-grained access control. For production applications, it is better to use a more secure and manageable method like OAuth or service principals, especially when calling Mosaic AI Model Serving endpoints.

Databricks secrets with the API token

Why it fails: Databricks secrets are a secure way to store credentials, but using them with a PAT still relies on a long-lived token. This approach is better than hardcoding but not as secure as short-lived OAuth tokens. Additionally, managing PAT rotation and permissions can be cumbersome. For service-to-service authentication, a service principal with OAuth is recommended.

Increase the temperature parameter to make the model's outputs more diverse and less likely to repeat harmful patterns.

Why it fails: Increasing temperature increases randomness, which can actually lead to more unpredictable and potentially harmful outputs. It does not mitigate harmful content; instead, it reduces coherence and may amplify biases. Temperature tuning is not a safety mechanism and can worsen the problem.

Implement a post-processing filter that scans model outputs for toxic or biased language and blocks them.

Why it fails: Post-processing filters are reactive and can catch harmful outputs, but they are not a primary mitigation. The question asks for safeguards to prevent generation; while post-processing can block, it does not prevent the model from generating harmful content in the first place. It is a complementary measure, not one of the two best approaches.

Use a larger model with more parameters, as larger models are inherently less biased.

Why it fails: Larger models are not inherently less biased; they can perpetuate or even amplify biases present in training data. Model size does not guarantee safety. This approach does not address the need for explicit safeguards and may increase cost and latency without solving the problem.

Create a Databricks Vector Search endpoint and pass its name to mlflow.evaluate as the model argument.

Why it fails: A Vector Search endpoint serves similarity queries but does not encapsulate the LLM generation step of the chain. mlflow.evaluate needs the full chain to produce both retrieval and generation metrics; passing only an endpoint name would either fail to load or score only retrieval, leaving the generation quality unmeasured for the prompt comparison.

Write a custom Python function that calls the chain and pass it to mlflow.evaluate as a pyfunc with a signature.

Why it fails: A pyfunc wrapper is a valid way to score arbitrary code, but it does not preserve the LangChain graph structure, so MLflow cannot automatically extract retriever documents or apply LangChain-aware metrics. The engineer would have to reimplement retrieval metrics manually, losing the built-in evaluator behavior that the scenario requires.

Register the chain in Unity Catalog as a model version and pass the fully qualified model name to mlflow.evaluate.

Why it fails: mlflow.evaluate expects either a Python callable, a local artifact path, or a logged MLflow model URI. A Unity Catalog model name is a registry reference used for governance and serving; passing it directly to evaluate does not give the evaluator a loadable chain, so no retriever outputs or metrics would be produced for this scenario.