CCNA ML Ops Questions

Blue-green deployment allows you to maintain a production endpoint while simultaneously deploying a new version. By using weighted traffic splitting, you can shift traffic gradually and monitor performance metrics. If the new version exhibits anomalies, you can instantly revert traffic to the stable version, ensuring high availability and minimizing the impact of potential production failures, which is critical for robust MLOps workflows.

Delta Lake's time-travel capability is the standard for ensuring data reproducibility in MLOps. By using 'VERSION AS OF' or 'TIMESTAMP AS OF', you can query the exact state of the training dataset at any point in time. This is critical for auditing, model retraining, and debugging, as it eliminates the ambiguity of changes made to the underlying data sources over time.

Model Registry webhooks are the critical component for automating governance. By triggering external CI/CD pipelines (like Jenkins or GitHub Actions) upon model version transitions (e.g., 'Pending' to 'Production'), organizations enforce quality gates. This ensures that only models validated by automated unit tests or performance benchmarks are promoted, reducing manual intervention and minimizing the risk of deploying underperforming models into production environments.

The Model Registry centralizes model management, providing a single source of truth for model lifecycle stages and version history. This is vital for MLOps, as it ensures that teams can track which models are in production, who approved them, and how they perform over time, enabling consistent deployment workflows and improved auditability for regulatory compliance and internal governance.

Feature Store keeps an offline table for training and an online table for serving. Updates to the offline table do not propagate automatically; a publish step materializes them into the online store. When the nightly refresh runs without a subsequent publish, the endpoint keeps reading the prior values, producing stale predictions for affected customers until the online store is refreshed.

Linking the deployment to a specific Model Registry version alias or version number is the only way to ensure immutability. When you serve a model, you should point to a specific version (e.g., version 5) rather than a dynamic alias like 'Production' if you need to guarantee that no subsequent transitions will change the behavior of the endpoint, thereby ensuring production stability.

MLflow Tracking is the primary API used for logging and querying machine learning experiments. It allows data scientists to record key information during the training phase, which is vital for experiment reproducibility and comparing performance across different model iterations. Without MLflow Tracking, maintaining visibility into the history of experiments would be manual, fragmented, and prone to losing critical context.

Model Serving endpoints can host multiple model versions and use traffic splitting to route requests. If a new version underperforms, you can instantly shift traffic back to a previous version by adjusting the split, achieving a fast rollback without redeployment. Other methods are manual, slow, or do not directly control the serving endpoint.

A Python wheel task is the purpose-built job task type for running packaged Python code: you point it at the wheel and a fully qualified entry point, and Databricks handles installation and invocation. Notebook-based installs or subprocess calls, and Spark submit tasks, do not provide the same reproducible, entry-point-driven execution.

MLflow separates metadata from artifacts: the backend store holds run and metric records, while the artifact store holds the model files. A deployment job that can read experiment metadata still needs its own credentials to the artifact location. Aligning the service principal's access, or exposing the path through Unity Catalog storage credentials and external locations, is what lets the artifact download succeed.

Traceability requires machine-readable links from the production model version back to the exact data and code. Recording the dataset version and Git commit on the source run, plus preserving the run's source notebook, gives auditors a verifiable chain. Descriptions, request logging, and permissions document or control access but do not establish provenance, so they cannot satisfy the requirement alone.

Grouping candidate runs in one experiment lets MLflow's search API rank and filter them by metric and tag in a single query, which is what an automated promotion step needs. Splitting experiments, overloading registry descriptions, or duplicating metrics elsewhere all break that integrated, queryable comparison.

Using a centralized MLflow Model Registry via Unity Catalog allows models to be shared across workspaces securely. This avoids the manual export/import of artifacts and ensures that lineage, tags, and versioning remain consistent. By managing access through Unity Catalog permissions, you ensure that only authorized environments can read or promote specific models, creating a unified, compliant, and highly scalable model deployment lifecycle.

Databricks Jobs with multi-task workflows is the built-in orchestration service that allows you to define dependencies, schedule triggers, and monitor execution of complex ML pipelines. It can be triggered by file arrival events, making it ideal for retraining when new data lands. Other options like MLflow Projects, Delta Live Tables, and Repos serve different purposes and lack native orchestration capabilities.

The correct approach is to archive the problematic Production version and promote the prior known-good version back to Production. This restores service using an already-validated artifact while maintaining full stage-transition history for audit and rollback traceability. Deleting versions, creating new models, or bypassing the registry all sacrifice governance or introduce unnecessary operational risk.

Inference tables are a Databricks Model Serving feature that automatically logs request and response payloads to a Delta table. This enables easy querying and monitoring for data drift without writing additional code. Other options either do not capture payloads automatically or are not designed for this purpose.

Using FeatureStoreClient.create_training_set ensures point-in-time correctness during training, and publishing the model with Feature Store ensures that inference uses the same feature lookups from the online store. Together, these actions guarantee that features are consistent between training and serving, which is critical for real-time fraud detection.

The Databricks Feature Store acts as a centralized repository for features. By using the same feature definitions and transformation logic for both training and serving, it eliminates the discrepancy known as training-serving skew. This ensures that the features fed to the model at inference time are calculated exactly as they were during training, maintaining consistent performance and model reliability.

Failing fast and sending alerts is the standard practice. Automated pipelines should include error handling that logs the failure cause to MLflow and triggers a notification via email or Slack. This ensures that data scientists can address the issue immediately without waiting for a manual audit of the pipeline, maintaining the overall health and reliability of the automated machine learning lifecycle.

Model Serving writes inference payloads to a Delta inference table when the feature is enabled, and that table survives endpoint deletion because it is a separate Unity Catalog object. Exporting the table before removing the endpoint preserves the historical request and response records required for compliance, whereas configuration snapshots or model artifacts do not contain traffic data.

The exhibit shows a hard-coded path beginning with '/dbfs/'. In Databricks, accessing DBFS via local file system paths can be inconsistent across different clusters or environments. The correct approach is to use the MLflow URI (e.g., 'runs:/...') to load models, as it abstracts the underlying storage location and ensures the artifact is resolved correctly regardless of the execution environment or storage configuration.

Databricks jobs support table-update triggers, letting a run start when the feature table changes. A multi-task job can then train the model and use a conditional task to register the version only when validation metrics beat the current production baseline, all within native orchestration.

Lakehouse Monitoring on the inference table is the only option that provides statistical monitoring of model inputs and outputs. It computes drift and profile metrics and, when ground truth labels are available, model quality metrics such as accuracy or RMSE. This makes it the correct mechanism for an SLO tied to prediction performance rather than simple endpoint availability.

MLflow Tracking is the core component of Databricks for experiment management. It provides a structured way to log and visualize the results of machine learning runs. Understanding this component is fundamental to MLOps because it allows teams to compare models, track performance over time, and ensure that every experiment is reproducible, which is essential for professional model development and auditing.

Wrapping preprocessing and the estimator in a single pyfunc model lets MLflow serialize both as one artifact. Batch scoring loads that artifact with the pyfunc loader, and Model Serving uses the same artifact, so feature transformations are guaranteed identical and training-serving skew is avoided without duplicating logic.

Unity Catalog acts as the centralized governance layer for all data and AI assets. It provides fine-grained access control and lineage tracking, allowing organizations to maintain visibility over which data is used by which models. By centralizing these assets, teams can collaborate safely while ensuring that compliance and auditing requirements are satisfied across the entire organization, regardless of the individual workspace where the work is performed.

Model-as-Code treats the entire deployment process—infrastructure, environment configuration, and code—as versioned artifacts. This allows for total reproducibility, where any production state can be rolled back or recreated using stored definitions. This practice is essential for enterprise MLOps, as it ensures that deployments are predictable, scalable, and audit-compliant, significantly reducing the risks associated with manual configuration changes in a production environment.

Inference logging captures the actual feature values sent to the model endpoint, and Lakehouse Monitoring provides built-in drift metrics by comparing these to a baseline profile. This combination enables automated, scalable drift detection without manual intervention. Other options either misrepresent platform features or fail to statistically compare distributions.

Databricks Lakehouse Monitoring provides automated insights into data quality and drift. By tracking statistical properties of incoming data against a baseline, it generates alerts when significant shifts are detected. This is a crucial component of MLOps, as it proactively identifies when a model's performance might degrade due to environmental changes, allowing teams to retrain or adjust models before they negatively impact business outcomes.

Enforcing approval workflows is crucial for compliance and risk management in MLOps. By using MLflow's permissions and stages, you can restrict who can promote models to 'Production'. Combining this with programmatic checks or external CI/CD gates ensures that a model cannot be deployed without the necessary authorization, preventing unauthorized or unvetted code from reaching production inference services.

Reproducibility across time requires controlling the environment, not just observing it. Pinning exact versions in the project's environment file guarantees the same dependencies are installed, and logging the environment specification with the run records the actual resolved environment for future rebuilding. Relying on runtime versions, notebook revisions, or autologging alone does not prevent dependency drift, so those approaches cannot ensure the same result six months later.

To ensure a custom library is available and reproducible in Databricks Model Serving, it must be included as a dependency when logging the MLflow model. Packaging the library as a wheel and adding it to the conda environment or requirements file ensures that Model Serving installs the exact version during deployment. This is the supported and recommended practice.

The error indicates that the deployment pipeline is attempting to access a model version that does not exist in the Model Registry. This usually happens when the pipeline triggers before the registration process completes or when a race condition occurs between the training job and the deployment job. Validating the existence of the model version before initiating deployment is essential for pipeline reliability.

When a deterministic model returns different outputs for the same input, the fastest path to the cause is to verify the two requests were truly identical and to confirm which model version handled each. A version transition during a rollout, or a payload difference such as field ordering or missing values, explains the symptom without any model retraining. Retraining, scaling, or changing flavors are premature and do not produce diagnostic evidence.

Auditing requires proof of data lineage, code versioning, and access control. Verifying these components ensures that the organization can prove how a model was built, what data was used, who approved it, and that the code was properly reviewed. This level of traceability is non-negotiable for regulated industries and essential for enterprise-grade MLOps maturity.

Implementing a policy-based deployment gate in your CI/CD pipeline ensures that governance requirements are met before code or models hit production. By querying the model's tags using the MLflow client before attempting a deployment, you create a hard stop for non-compliant models. This is a critical security and compliance practice, preventing unauthorized or untested models from being served to end-users or critical business processes.

The Gold layer contains highly processed, business-level data that is ready for consumption by downstream ML models or analytical applications. By ensuring data is clean, aggregated, and validated at this stage, MLOps teams can rely on high-quality features for training, which directly improves model performance and reduces the complexity of the feature engineering step in the training pipeline.

The MLflow UI experiment page is designed for comparing runs: it displays all runs in a table, supports sorting by any metric, and allows selecting multiple runs for side-by-side comparison of parameters, metrics, and artifacts. Manual recording, CSV export, or custom API scripts are less efficient and unnecessary for this common task.

Consistency between online and offline features is achieved by using the same pipeline to publish to both stores and by using point-in-time lookups during training. These practices ensure that the transformations and temporal alignment match between training and inference. Manual copying or direct Delta queries do not provide the required consistency or performance.

The requirement is automatic reload when a model version enters the 'Production' stage. Databricks Model Serving supports this directly by referencing the registered model name and stage in the endpoint configuration. Other approaches either bypass the registry, require manual redeployment, or rely on mechanisms the platform does not provide, making them unsuitable for a managed real-time endpoint with stage-driven updates.

To conditionally transition a model version based on a metric, you need to programmatically retrieve the metric and then call the MLflow Client's transition method. This allows you to enforce a threshold and fail the job if the condition is not met. Webhooks and REST API calls lack the ability to evaluate metrics before transitioning, and setting the stage during registration does not incorporate a conditional check.

An MLOps engineer's role is to ensure stability, performance, and compliance. This includes monitoring model performance to detect drift, managing the CI/CD pipeline for automated deployments, and ensuring security via robust access controls. By balancing these tasks, they ensure that the machine learning system remains reliable and valuable to the business over time, effectively bridging the gap between development and operations.

The model's feature spec records the versions of the feature tables used during training and logging. When a new feature table version is published, existing models still reference the old version, leading to stale values at inference. To resolve this, the model must be re-logged with an updated feature spec that includes the new version, and the endpoint must be updated with the new model version.

Effective monitoring of a production model requires capturing inference data and analyzing it for drift and performance. Enabling inference logging provides the raw data, while scheduling a job to compute drift metrics against the training set enables detection of data drift. Together, these actions allow the engineer to identify when the model's input distribution diverges from training, which is a key signal of degradation.

Databricks Container Services allows users to define custom Docker images as the execution environment. This is the only way to handle complex system-level dependencies like C++ libraries that cannot be installed via standard pip or conda commands. By using custom containers, teams ensure that the training and inference environments are identical and include all necessary system-level components.

MLflow Tracking is the dedicated component designed to record every experiment's parameters, metrics, and code state. By capturing these elements, data scientists can compare different iterations of a model to identify the best-performing configuration. This visibility is vital in MLOps, as it creates an audit trail of how a final model was derived, supporting reproducibility and informed decision-making before promoting a model to the registry.

To ensure the model uses the correct feature values at inference time, you must log the model with the feature store lookup tables and specify the feature table names. This enables automatic feature retrieval, maintaining consistency between training and serving and ensuring that daily updates are reflected in predictions.

MLflow Projects are designed to encapsulate dependencies in a conda.yaml, and running them with conda enabled creates an isolated environment that installs the specified versions. This avoids conflicts with shared cluster libraries and preserves reproducibility. Flags controlling experiment names, global installs, or notebook conversions do not provide the same isolation or reliability.

A Databricks Job provides the orchestration to run training and validation when new data arrives, and a Model Serving endpoint is required to deploy the validated model for real-time predictions. Together, they form the backbone of an automated retraining and deployment pipeline. MLflow Tracking, feature stores, and Repos are valuable but not strictly essential for the pipeline to function.

Linking models to specific Git commits and tracking the environment dependencies via Conda or pip requirements files are essential for reproducibility. These steps ensure that when a model is redeployed, the exact code version and environment can be recreated, fulfilling the core MLOps requirement of auditability and consistent model behavior across different computing environments.

Databricks Feature Store enables point-in-time joins to avoid data leakage during training and supports publishing to low-latency online stores (e.g., DynamoDB or Cosmos DB). By using a centralized feature store, teams ensure consistency between training and serving. Point-in-time lookups are crucial to ensure that features used for predictions reflect exactly what was known at that specific moment, preventing bias.

Databricks Model Serving provides serverless, auto-scaling inference endpoints. By default, it manages the underlying infrastructure, scaling the number of replicas based on real-time request volume. This is a key MLOps requirement to ensure high availability and performance without manual intervention, allowing teams to handle unpredictable traffic spikes without over-provisioning and incurring unnecessary costs during low-traffic periods.

Model Serving supports rolling updates that keep the endpoint live while a new model version loads, and the previous configuration remains restorable. Repointing the endpoint to the newer version and letting the rolling update run achieves zero-downtime promotion with a straightforward rollback. Deleting the old version or duplicating endpoints introduces risk or cost without improving the outcome.