AI Operations practice questions

Namespace policy

Why it fails: A namespace policy governs pod security, quotas and admission rules across a namespace, not per-step resource requests. It is tempting because namespaces constrain compute, but Kubeflow step resources are declared in the pipeline specification's component or container resource fields, which the compiler translates into pod specs.

Kubeconfig file

Why it fails: The kubeconfig file stores cluster endpoints, contexts and user credentials for kubectl access, containing no pipeline step definitions. It is tempting because it configures Kubernetes interaction, but resource requests for individual Kubeflow steps live in the pipeline YAML's component specification, not in client connection settings.

Service mesh config

Why it fails: Service mesh configuration manages traffic routing, mTLS and telemetry between services, not pod resource requests. It is tempting because meshes influence workload behaviour, but CPU and memory requests for Kubeflow steps are declared in the pipeline specification's component definitions, which Kubernetes reads when scheduling each step's pod.

Environment variable list

Why it fails: Too verbose for standard lineage.

User IP address

Why it fails: Irrelevant to lineage tracking.

Compute cluster name

Why it fails: Does not prove model provenance.

Custom prediction routines

Why it fails: Used for preprocessing, not monitoring skew.

Artifact metadata locks

Why it fails: Prevents accidental overwrites.

Pipeline concurrency limits

Why it fails: Controls resource usage, not skew.

To manage database indexes

Why it fails: Champion-Challenger compares a challenger model's live performance against the champion to decide promotion; database index management is unrelated to model deployment. It is tempting because MLOps pipelines do touch data stores, but index tuning belongs to database administration, not to a traffic-splitting deployment pattern.

To compress model weights

Why it fails: Champion-Challenger splits live traffic between champion and challenger models to compare performance before promotion; it does not alter model weights. It is tempting because model size matters in deployment, but compression is achieved through quantisation or pruning, not through this deployment pattern.

To increase training speed

Why it fails: Champion-Challenger routes a portion of live traffic to a challenger model and compares its performance against the champion, governing model promotion; it does not affect training throughput. It is tempting because faster training is a legitimate MLOps goal, achieved through distributed training or hardware acceleration instead.

Throughput (TPS)

Why it fails: Throughput counts requests served per second, revealing nothing about whether outputs are grounded in retrieved context. It is tempting because throughput monitoring is genuinely useful for capacity planning and autoscaling decisions, where it would be the right metric.

Token generation latency

Why it fails: Latency measures response speed, not factual accuracy; a fluent, fast answer can still be ungrounded. It is tempting because latency monitoring is genuinely useful for detecting performance degradation and capacity problems, where it would be the right metric.

Error code distribution

Why it fails: Error codes capture transport and API failures, not semantically plausible but ungrounded content, which returns HTTP 200. It is tempting because error distribution monitoring is genuinely useful for detecting service faults and integration breakage, where it would be the right metric.

Memory footprint

Why it fails: Memory footprint tracks resource consumption, not whether generated claims are supported by source context. It is tempting because memory monitoring is genuinely useful for detecting leaks and sizing inference hosts, where it would be the right metric.

Cosine Similarity

Why it fails: Cosine Similarity measures distance between text or embedding vectors, which suits natural language monitoring rather than structured tabular feature attribution. It is tempting because SageMaker Model Monitor does compute it, but that applies to text-based bias and drift checks, whereas structured attribution drift relies on SHAP values.

Mean Square Error

Why it fails: Mean Square Error measures regression prediction accuracy against ground-truth labels, not shifts in feature attribution distributions. It is tempting because MSE is a standard SageMaker Model Monitor metric, but that metric belongs to model quality monitoring, whereas feature attribution drift requires comparing SHAP-based attribution baselines.

Precision-Recall AUC

Why it fails: Precision-Recall AUC evaluates classification performance against labelled outcomes, revealing nothing about changes in feature attribution distributions. It is tempting because it is a recognised Model Monitor metric, but it sits under model quality monitoring, while feature attribution drift compares SHAP-value baselines for structured data.

Instance type selection

Why it fails: Instance type selection chooses the compute hardware for a training job; it captures no record of whether the input data was encrypted, so auditors gain no verifiable evidence. It is tempting because instance choice is a genuine training configuration decision, but it is correct only when the requirement is compute capacity or cost, not data-encryption provenance.

Model registry labels

Why it fails: Model registry labels are free-form metadata attached to model versions for organisation and approval workflows; they record no cryptographic provenance of the training data, so auditors cannot verify encryption from them. They are tempting because the registry does track model lineage, but only for artefacts, not the encryption state of underlying datasets.

Notebook instance tags

Why it fails: Notebook instance tags are key-value labels applied to the development compute resource, not to training jobs or their input data, so they evidence nothing about encryption during training. They are tempting because tagging supports cost allocation and resource governance, but that is their purpose, not audit verification of training-data encryption.

Environment YAML files

Why it fails: These define dependencies, not permissions.

Compute Instance logs

Why it fails: Compute instances do not manage registry access.

Data labeling projects

Why it fails: Unrelated to registry security.

Encrypting network traffic

Why it fails: Orchestrators schedule, sequence and monitor pipeline steps such as training and data preparation; they do not encrypt network traffic, which TLS and VPN layers handle. It tempts because pipelines move data between services, and would be correct if the requirement were securing data in transit between components.

Writing code for the model

Why it fails: Orchestrators coordinate and schedule existing pipeline steps; they do not author model code, which data scientists write in notebooks or scripts. It tempts because pipelines execute training code, and would be correct if the tool were a development environment such as Jupyter or an IDE rather than a workflow engine.

Optimizing SQL queries

Why it fails: Handled by DB engine.

Computer Intelligence and Cloud Data

Why it fails: CI/CD expands to Continuous Integration and Continuous Delivery/Deployment, not Computer Intelligence and Cloud Data. It is tempting because each word sounds plausible in an AI context, yet none matches the DevOps terminology describing automated build, test and release pipelines for AI Operations.

Continuous Inference and Cloud Deployment

Why it fails: CI/CD stands for Continuous Integration and Continuous Delivery/Deployment; inference and cloud deployment are not the terms. It is tempting because AI Operations genuinely involves inference serving and cloud deployment, but the acronym's first C refers to integrating code changes, not running model inference.

Code Integration and Cloud Delivery

Why it fails: CI/CD denotes Continuous Integration and Continuous Delivery/Deployment, not Code Integration and Cloud Delivery. The option is tempting because it echoes genuine DevOps vocabulary — code, cloud, delivery — but it misnames the integration and continuity concepts that define the practice in AI Operations pipelines.

Manual code refactoring

Why it fails: Model observability relies on runtime instrumentation — logging predictions, monitoring drift, latency and data quality — not on rewriting source code. Manual refactoring alters model or pipeline code before deployment and produces no telemetry about live behaviour. Refactoring would be the right activity when remediating technical debt or improving maintainability, not for observability.

Database backup frequency

Why it fails: Database backup frequency concerns durability and recovery of stored data, not the runtime behaviour of a deployed model. Observability requires capturing inference metrics, input drift and prediction distributions from the serving path. Backup scheduling would be the correct control when defining recovery point objectives for the feature store or training data.

Developer salary audits

Why it fails: Salary audits reveal nothing about a deployed model's runtime behaviour, so they cannot surface drift, latency or token-level metrics. They are tempting because workforce cost analysis belongs to HR budgeting, where auditing developer pay against market bands would indeed be the right exercise.

Internal employee passwords

Why it fails: Model Cards disclose model purpose, data provenance, metrics and limitations; employee passwords are credentials and must never appear in any documentation. It is tempting because access control is a genuine model risk concern, and in a security review credential management would be the correct topic—but it is not a Model Card field.

Cloud service provider revenue

Why it fails: Model Cards document a model's intended use, training data, performance and limitations; provider revenue is commercial information with no bearing on model risk. It is tempting because vendor financials appear in procurement due diligence, where revenue would legitimately matter—but that is a supplier assessment, not a Model Card disclosure.

During feature engineering

Why it fails: Feature engineering produces the training data, not a trained model, so there is nothing to validate at that stage. It is tempting because data quality strongly influences model outcomes, and validation there would be correct for feature-distribution or data-drift checks rather than model performance gates.

Before source code check-in

Why it fails: Model validation requires a trained candidate model and its evaluation data, neither of which exists before source code check-in. It is tempting because early quality gates catch defects cheaply, which would be correct for static analysis or lint checks rather than model evaluation.

After deployment

Why it fails: Placing model validation after deployment lets a faulty model reach production before any check runs, so it cannot block promotion. It is tempting because post-deployment monitoring catches drift, but the validation gate must sit before deployment to prevent bad models shipping.

Use absolute file paths

Why it fails: Absolute paths break reproducibility across machines and do not make data immutable; DVC tracks content by hash, so paths are irrelevant to audit integrity. It tempts because path configuration feels like a versioning concern, and absolute paths would be correct where a fixed local mount must be referenced explicitly.

Disable data caching

Why it fails: Disabling caching removes DVC's local link between workspace files and the cache, so nothing anchors a commit to specific content; immutability comes from committing the .dvc pointer file and pushing to remote storage. Caching is tempting because it speeds up checkout, but it governs performance, not version integrity.

Store data on local drive

Why it fails: A local drive gives no remote copy, so the audit trail exists only on one machine and cannot be verified or restored elsewhere; immutability requires committing .dvc pointer files and pushing data to remote storage. Local storage suits quick single-machine experiments, not auditable versioning.

Kubernetes manifest

Why it fails: A Kubernetes manifest deploys and orchestrates containers; it generates no explanation artefacts. It is tempting because the endpoint runs on Kubernetes, but the missing logs come from an explainability library such as SHAP or LIME that must be invoked in the inference code, not from the manifest.

TensorBoard logging

Why it fails: TensorBoard logging records training metrics such as loss and accuracy; it does not produce per-inference explanation output. It is tempting because it is a familiar logging tool, but explanation logs require an explainability library like SHAP or LIME integrated into the serving code, which TensorBoard cannot supply.

Prometheus exporter

Why it fails: Prometheus exporters publish numeric metrics for monitoring and alerting; they do not generate model explanation artefacts. Tempting because it is a standard observability component, but explanation logs come from an interpretability library such as SHAP or Azure Machine Learning's responsible AI tooling, so its absence is unrelated.

The training data was leaked

Why it fails: Data leakage is a training or pipeline defect where evaluation data contaminates training, not a post-deployment shift in the input-target relationship. It is tempting because both surface as unexplained accuracy loss, and leakage would be correct if the audit found test samples inside the training set.

The model is overfitted

Why it fails: Overfitting is a training-time generalisation failure where the model memorises training examples; concept drift instead describes the statistical relationship between inputs and target changing after deployment. Overfitting is tempting because both degrade live accuracy, and it would be the answer if validation loss diverged sharply from training loss.

The model is running too fast

Why it fails: Concept drift describes the statistical relationship between model inputs and outputs changing over time, degrading prediction accuracy. Execution speed is unrelated to drift. The option is tempting because drift can accompany infrastructure changes, but latency is a performance metric, not a data-distribution shift.

Marketing Intern

Why it fails: A marketing intern lacks the organisational authority and accountability to approve or restrict AI systems. It tempts because interns may use generative AI tools daily, giving practical familiarity, but governance oversight requires decision rights held by accountable senior roles.

Janitorial Staff

Why it fails: Janitorial staff hold no authority over model risk, data handling or deployment decisions, so they cannot enforce governance controls. It tempts because facilities personnel may observe physical access to AI hardware, which is a physical-security concern, not AI oversight.

Employee vacation schedule

Why it fails: An employee vacation schedule records staffing availability, not a model's lineage, version, training dataset or performance metrics, so it fails the metadata requirement. Registries do hold descriptive attributes, which makes any business record seem plausible, yet only attributes describing the model artefact itself qualify.

Cafeteria menu

Why it fails: A cafeteria menu describes nothing about a model's provenance, training data, hyperparameters or evaluation metrics, so it cannot be registered as model metadata. It is tempting because registries do store arbitrary descriptive tags, but those tags must characterise the model artefact itself, not unrelated operational content.

Office Wi-Fi password

Why it fails: Model metadata describes the model itself, such as version, training dataset, hyperparameters, and evaluation metrics; a Wi-Fi password is unrelated infrastructure credential data. It is tempting because registries track artefacts and credentials, but network access secrets belong in a secrets manager, not model metadata.

Low training accuracy, low validation accuracy

Why it fails: Overfitting produces high training accuracy alongside poor validation accuracy, because the model memorises training noise. Low values on both indicate underfitting instead, where the model has learned too little from the training data to generalise or fit it.

Model crashes during training

Why it fails: A crash is a runtime or resource failure, not an overfitting symptom; overfitting shows as low training loss with high validation loss. It is tempting because training instability feels like a training-phase fault, and crashing would be the expected symptom of excessive model capacity or an unstable learning rate, not memorisation.

Model is too small

Why it fails: Model size is a capacity choice, not a training symptom; an undersized model underfits, producing high training and validation error. It is tempting because capacity relates to overfitting, but overfitting is diagnosed from the training-versus-validation loss gap, which a small model does not produce.

Model accuracy curve

Why it fails: A DAG describes the ordered dependency graph of pipeline tasks or model steps, not a performance metric. Accuracy curves are plotted from evaluation results such as loss or ROC data. A DAG would be the answer when the question asks how task dependencies and execution order are represented.

The database schema

Why it fails: A DAG encodes task dependencies and execution order between pipeline steps, not table relationships or column definitions. Confusing it with a database schema is tempting because both are structural diagrams, but a schema describes persistent data storage, whereas the DAG describes orchestration flow.

The network architecture

Why it fails: A DAG captures task dependencies and execution ordering across pipeline stages, not layer counts, neurons or activation functions. Network architecture is tempting because both are graphs, but topology describes the model's computation, while the DAG describes orchestration of jobs around it.