ML Ops practice questions

Manually add the git hash as a tag to every MLflow run.

Why it fails: Manual tagging is error-prone and tedious. It relies on developer discipline, which is not sustainable in a professional team. Databricks provides built-in mechanisms to handle this automatically, which should always be preferred over manual, manual, or custom-scripted solutions that might fail if developers forget to tag.

Write a custom script to query the git status and log it as a parameter.

Why it fails: Custom scripts increase maintenance overhead and are unnecessary given built-in platform features. They add complexity to the training code and are more likely to break during platform updates. Utilizing platform-native integration is a best practice for clean, maintainable, and robust MLOps code in a Databricks ecosystem.

Upgrade to the Databricks Premium tier to enable git logging.

Why it fails: Git integration via Repos is a core capability and not restricted by tier in this manner. Claiming it requires a premium upgrade is factually incorrect and ignores the fundamental architecture of the platform. Focus should be on correctly configuring existing tools rather than changing subscription plans.

The model version does not exist in the registry.

Why it fails: If the model version did not exist, the error would specifically state 'RESOURCE_DOES_NOT_EXIST: Model version not found'. The error clearly indicates that the issue is related to the current stage of the model, implying the version exists but is in a state that prevents deletion.

The user lacks permissions to delete models from the registry.

Why it fails: Permission errors would return a 'PERMISSION_DENIED' status. Since the error is a 'RESOURCE_DOES_NOT_EXIST' related to the stage, it confirms the operation is technically blocked by the workflow rules, not by the user's specific access rights, making this explanation technically inaccurate for the provided log.

The model name is incorrectly specified in the API call.

Why it fails: If the model name were incorrect, the registry would fail to locate the model entirely, resulting in a different error message. The presence of the specific version number and stage in the error log indicates the registry successfully located the model but failed the state validation check.

Grant all data scientists 'Manage' permissions on the Model Registry to allow direct movement to production.

Why it fails: Granting 'Manage' permissions to all users violates the principle of least privilege. This approach creates a security vulnerability where unverified models could bypass quality gates, leading to potential production outages or performance regressions. Governance should be centralized to maintain strict control over model artifacts.

Manually update the model version alias to 'production' via the Databricks UI without external pipeline triggers.

Why it fails: Manual updates lack an audit trail and do not integrate with automated testing frameworks. Relying on manual intervention is prone to human error and prevents the implementation of standardized deployment patterns, making it difficult to maintain consistent quality assurance across different model versions.

Implement a Python script that uses the MLflow client to force move models to the 'production' stage based on time-based schedules.

Why it fails: Time-based promotions ignore the health and performance status of the model, which is dangerous in ML workflows. A model should only be deployed based on quality and performance metrics, not an arbitrary schedule. This ignores the necessity of validation and stakeholder approval entirely.

Use separate transformation logic for the online store to optimize for lower latency.

Why it fails: Using different logic for offline and online transformations introduces training-serving skew, which is a major source of model performance degradation. The same feature engineering code must be used to guarantee that the features consumed by the model at inference match the features used during training.

Exclusively store feature values in memory to maximize retrieval speed for online serving.

Why it fails: While in-memory storage is fast, it is not scalable for large feature sets and lacks persistence. The Databricks Feature Store supports dedicated online stores like Azure Cosmos DB or Amazon DynamoDB, which are designed for high-availability, low-latency, and persistent storage, unlike volatile in-memory caches.

Avoid using Delta tables for the offline store to improve throughput performance.

Why it fails: Delta tables are highly optimized for batch processing and large-scale data retrieval, making them the industry standard for offline feature storage. Avoiding them would remove the benefits of ACID compliance, time travel, and schema enforcement, which are essential for maintaining accurate and reproducible machine learning datasets.

The model version is missing a 'production' tag.

Why it fails: Tags are metadata and do not control the workflow state of the MLflow model registry. While tags are useful for searching and filtering, the deployment trigger relies on the lifecycle stage (e.g., 'Staging' or 'Production'), not the presence of a specific key-value pair in the tags.

The validation score of 0.88 is too low to trigger the deployment script.

Why it fails: The validation score is simply a string tag in the metadata. Unless there is custom logic inside the CI/CD job explicitly checking the value of the tag, the registry itself does not filter deployments based on tag values during the initial model registration process.

The 'team' tag is not authorized to trigger production deployments.

Why it fails: Tags are not authorization mechanisms in Databricks or MLflow. RBAC (Role-Based Access Control) is handled via Unity Catalog or workspace permissions, not via tag values. Therefore, the 'team' tag has no impact on whether a deployment script can access or deploy the model version.

Hard-coding the production credentials directly into the notebook for easy access.

Why it fails: Hard-coding credentials is a severe security violation. It exposes sensitive information, making it accessible to anyone with access to the code. Proper MLOps requires using secure credential management like Databricks Secrets to handle authentication tokens and connection strings in a secure, encrypted manner.

Deploying the model to production and monitoring it for 24 hours to validate success.

Why it fails: Deploying an unvalidated model to production to test it is known as 'testing in production' and is highly risky. Validation should always happen in a non-production environment. Using production as a testing ground can lead to degraded user experiences and potential business impact if the model fails.

Set LOG_LEVEL to 'DEBUG' to get more information about the timeout.

Why it fails: Increasing the log level to 'DEBUG' will provide more information about the process but will not resolve the underlying performance issue. It may actually degrade performance further by increasing IO overhead. The root cause is the timeout threshold, not a lack of diagnostic information.

Disable ENABLE_AUTO_REGISTRATION to prevent the model from being deployed to PROD.

Why it fails: Disabling registration does not fix the performance issue; it only stops the deployment. The goal is to make the model work, not to prevent it from reaching the production environment. This action is unrelated to the timeout error occurring during model serving.

Change the DEPLOY_ENVIRONMENT to 'STAGING' to bypass production constraints.

Why it fails: Changing the environment is not a solution to a technical performance limitation. The same timeout will likely persist in staging unless the underlying configuration (the timeout limit) is adjusted. Bypassing production constraints does not resolve the root cause of the RequestTimeoutException during inference.

Use the MLflow UI to manually check the library versions for every model.

Why it fails: Manual checking is slow and impossible to enforce consistently. It does not provide the automated gatekeeping necessary for a modern MLOps pipeline. Security checks should be automated during the deployment process, not left to human inspection, which is inherently inconsistent and prone to missing vulnerabilities.

Rely on the default environment provided by Databricks, which is always secure.

Why it fails: While Databricks provides managed runtimes, the specific dependencies (libraries) added by users are not automatically audited for all possible vulnerabilities. Relying solely on the platform's base environment is insufficient for a rigorous security strategy that requires checking user-defined package dependencies.

Configure the Model Registry to automatically delete models that use outdated libraries.

Why it fails: Automatically deleting models is destructive and lacks nuance. It could result in the loss of valuable work and disruption of services. Proper MLOps involves scanning and flagging, not automated deletion, which would be extremely detrimental to the workflow and overall team productivity.

Update the existing endpoint configuration directly and wait for the auto-restart process.

Why it fails: Directly updating the existing endpoint configuration triggers a rolling update, but it does not provide the same risk mitigation as a Blue-Green deployment. If the new library causes runtime failures, your production environment will be impacted until a manual rollback is initiated, resulting in unacceptable downtime.

Delete the endpoint and recreate it with the new configuration during a scheduled maintenance window.

Why it fails: Deleting the endpoint creates a complete service outage. This violates modern MLOps requirements for high availability. While simple to implement, it is not a professional strategy for production systems where clients expect continuous access to inference services, leading to loss of data and potential business impact.

Deploy the model as a batch inference job instead of a real-time endpoint.

Why it fails: Batch inference is architecturally different and does not solve the problem of real-time service availability. If the client expects low-latency inference, batch jobs will fail to meet the requirements and do not provide a mechanism for continuous traffic serving during environment updates.

Delta Sharing for model distribution.

Why it fails: Delta Sharing is designed for secure cross-organization data sharing. It does not provide model lineage or artifact tracking capabilities. While useful for accessing external datasets, it is not a governance tool for auditing model development lifecycles or tracking the provenance of model versions within a workspace.

MLflow Experiment tracking.

Why it fails: MLflow Experiment tracking manages model runs, parameters, and metrics, but it does not provide the enterprise-level lineage mapping across the entire data estate. While it records input data signatures, it lacks the cross-workspace data governance and auditability features inherent to Unity Catalog's lineage capabilities.

Feature Store workspace-level tags.

Why it fails: Workspace-level tags are manual metadata and do not constitute automated lineage. They are prone to human error and lack the granular, verifiable audit trails provided by the automated lineage features of Unity Catalog, making them insufficient for strict enterprise governance requirements for ML projects.

The model version is already in the 'Archived' state.

Why it fails: If the model were archived, the error would typically indicate an invalid state transition rather than a permission error. The 'PERMISSION_DENIED' code is specific to authorization failure, implying the user lacks the necessary privilege to modify the model registry state regardless of the version's current status.

The model depends on an external library that is not installed.

Why it fails: Library dependencies impact model execution, not the metadata transition in the registry. An issue with libraries would result in a model loading failure or runtime error during inference, not a 'PERMISSION_DENIED' message when attempting to register or change the status of a model version.

The Unity Catalog metastore is currently offline.

Why it fails: If the metastore were offline, the error would manifest as a connectivity or timeout issue, such as 'METASTORE_UNREACHABLE'. A 'PERMISSION_DENIED' error indicates successful communication with the authorization service, confirming the user is authenticated but lacks the specific privilege required to perform the requested model update action.

The model must be stored as a raw CSV file.

Why it fails: Model serving requires serialized formats like MLflow's native pickle, ONNX, or MLmodel formats. CSV is a data format, not a model serialization format. Attempting to deploy a CSV file as a model will fail because the serving engine cannot execute logic from a static data file.

The model must be in the 'Archived' registry state.

Why it fails: Models in the 'Archived' state are generally unavailable for active serving. The serving service typically requires models to be in 'Staging' or 'Production' states. Archiving is intended for models that are no longer in use, and attempting to serve an archived model will result in deployment failure.

Deploy two separate endpoints and use a load balancer to split traffic.

Why it fails: Splitting traffic means some users get the new model. Shadowing requires sending the *same* request to both. A load balancer split would be an A/B test or canary deployment, not a shadow deployment. Shadow deployment specifically requires processing the same input in parallel without impacting the production user experience.

Run two inference jobs in a single notebook periodically.

Why it fails: This approach is for batch inference, not real-time shadow deployment. Shadow deployments refer to real-time inference services. Running periodic notebooks does not handle live incoming requests and therefore cannot serve as a shadow deployment mechanism for real-time model validation against production traffic patterns in real-time.

Manually copy request data from the production logs for re-testing.

Why it fails: This is a reactive and manual approach, not a true shadow deployment strategy. It is not real-time and prone to errors. True shadow deployments are automated, providing live, continuous assessment of the candidate model performance against the same requests that the production model is currently processing.

Package the model with a custom pyfunc signature and deploy it to a CPU-based Serving endpoint using standard instance configurations.

Why it fails: Deploying deep learning models on CPU instances severely degrades inference performance and limits throughput. PyTorch inference workloads benefit significantly from hardware acceleration provided by GPU instance types configured directly on the Databricks Serving endpoint.

Export the model weights to an external S3 bucket and invoke them via an external API gateway bypassing Databricks serving infrastructure.

Why it fails: Externalizing inference outside Databricks introduces unnecessary network latency, security complexity, and operational overhead. Managing endpoints natively within Databricks ensures unified governance, access control, and seamless integration with Unity Catalog model registry.

Run an interactive notebook continuously in production that reads requests from a Delta table and writes predictions back periodically.

Why it fails: Interactive notebooks running continuously are unsuitable for real-time model serving due to high latency and lack of auto-scaling. Model serving endpoints provide REST APIs with millisecond response times and robust enterprise reliability.

Manually update the model version status in the MLflow UI after each successful test run.

Why it fails: Manual processes lack scalability and auditability in production environments. Requiring human intervention increases the risk of operational errors and does not integrate into automated CI/CD workflows, which are essential for maintaining consistent deployment standards and reducing the time-to-market for model updates in professional Databricks machine learning projects.

Add a tag to the model version using the MLflow SDK and check for the tag in a daily scheduled job.

Why it fails: Relying on scheduled polling jobs introduces latency and inefficiency compared to event-driven architectures. Webhooks provide real-time updates, whereas polling requires constant compute resources to check for tag changes, making it a sub-optimal choice for high-frequency or high-reliability MLOps workflows in production Databricks environments.

Hardcode the transition logic inside the training notebook using the transition_model_version_stage method.

Why it fails: Hardcoding model transition logic within training code tightly couples training and deployment, violating separation of concerns. This approach is difficult to manage as it requires modifying notebooks for environment changes, hindering scalability and making the CI/CD pipeline brittle and prone to failure during automated execution cycles.

Only store the final serialized model object in the MLflow Model Registry to save storage.

Why it fails: Storing only the serialized object provides no context about the environment, dependencies, or training lineage. This makes debugging impossible if the model fails in production, as you cannot determine the underlying causes. Comprehensive logging, including metadata and environment snapshots, is required for professional-grade MLOps model governance.

Rely on the default Databricks Runtime version for all training and inference tasks.

Why it fails: Databricks Runtimes change periodically. Depending on default versions without pinning specific configurations leads to environment drift. Reproducibility mandates that you explicitly define and log the environment context, as relying on global defaults makes it impossible to guarantee that a model trained months ago will execute correctly today.

Delete old model versions immediately to keep the registry clean and organized.

Why it fails: Deleting model versions destroys lineage and audit trails. In regulated industries, preserving the history of all model versions is a compliance requirement. Instead of deletion, use lifecycle stages like 'Archived' to manage the registry effectively while maintaining the necessary historical data for audits and potential rollback scenarios.

MLflow Model Registry

Why it fails: The Model Registry is for versioning and managing the lifecycle stages of a model (e.g., Staging, Production). It does not contain tools for analyzing input data or detecting drift in data distributions. It acts as a repository for model artifacts rather than an active monitoring engine for data quality.

Databricks Feature Store

Why it fails: The Feature Store is designed to ensure consistency between training and serving features. While it keeps track of feature lineage, it does not inherently monitor for data drift in production pipelines. It lacks the statistical analysis capabilities required to compare real-time inference data distributions against historical training distributions.

MLflow Tracking

Why it fails: MLflow Tracking logs training parameters and performance metrics for individual experiment runs. It does not provide real-time monitoring of inference datasets or capabilities for identifying statistical drift in production environments. It is a retrospective tool for model development, not an operational monitor for live data pipelines.

Always run CI/CD jobs using a shared, long-running interactive cluster to save costs.

Why it fails: Shared clusters are prone to environment contamination and permission issues. CI/CD pipelines should always use transient, job-scoped clusters to ensure the environment is clean and isolated. This prevents dependencies from one run affecting another, which is critical for the reproducibility and reliability of automated testing and deployment processes.

Perform automated integration tests on the production model before moving it to staging.

Why it fails: Testing should happen in an isolated environment that mirrors production before deployment. Running tests on the production model directly is dangerous, as it can disrupt active services and affect live users. A proper MLOps pipeline flows from dev to staging/QA, and only then to production after verification.

The model version is incompatible with the server's runtime version.

Why it fails: A version incompatibility usually triggers a specific 400 or 500 error related to dependency resolution or library errors. A 503 error is specifically a 'Service Unavailable' status, which in this context indicates that the infrastructure is still in the process of starting or scaling, not a configuration mismatch.

The user does not have permission to access the model endpoint.

Why it fails: Lack of permission results in a 403 Forbidden error, not a 503 Service Unavailable error. The 503 code indicates that the service itself is temporarily unable to handle the request due to infrastructure state, rather than an authentication or authorization failure triggered by the user's specific identity.

The model artifact has been deleted from the MLflow Model Registry.

Why it fails: If the model artifact was missing, the endpoint would fail to start entirely and would report a permanent error (like a 404 or a specific deployment failure), not a temporary 503 error. 503 errors are transient and are typically related to the operational health and scaling status of the service.

It automatically generates the raw data ingestion pipelines from external RDBMS sources.

Why it fails: The Feature Store does not automatically build ingestion pipelines for external databases. Data engineers must still construct the ETL processes to bring data into the Delta tables that the Feature Store consumes. The store provides the management and serving layer, not the source ingestion logic.

It guarantees that all models deployed will automatically achieve 99.9% prediction accuracy.

Why it fails: Feature Stores improve data consistency and operational efficiency, but they cannot guarantee model accuracy. Accuracy is a function of feature quality, algorithm choice, and hyperparameters. No infrastructure tool can replace the necessity of rigorous model evaluation and data science work required to optimize predictive performance.

It replaces the need for MLflow by handling all experiment tracking.

Why it fails: The Feature Store and MLflow serve different purposes. MLflow is for tracking parameters, metrics, and models, while the Feature Store manages feature definitions and serving. They are designed to work together in the Databricks ecosystem; the Feature Store does not have the capability to replace MLflow's experiment tracking.