AAIA AI Operations — 15 Questions

Question 1

What is the primary purpose of a 'Champion-Challenger' deployment pattern in MLOps?

Accepted Answer

To safely test a new model vs production. It allows safe A/B testing of a new model version against the current production model.

Answer

To manage database indexes

Answer

To compress model weights

Answer

To increase training speed

Question 2

When auditing Kubeflow pipelines, which configuration file is used to define the resource requests for individual pipeline steps?

Accepted Answer

resource_spec in the SDK. The 'pipeline.yaml' or the step-specific resource configuration in the SDK defines CPU/Memory limits.

Answer

Namespace policy

Answer

Kubeconfig file

Answer

Service mesh config

Question 3

For a production LLM deployment, which operational metric is most critical to monitor to identify 'hallucination' or grounding failure in real-time?

Accepted Answer

Context faithfulness score. Retrieval metrics, specifically faithfulness or answer relevance, indicate if the model is hallucinating outside of retrieved context.

Answer

Throughput (TPS)

Answer

Token generation latency

Answer

Error code distribution

Answer

Memory footprint

Question 4

An organization uses MLflow for model tracking. To ensure auditability of model lineage, what must be captured in the 'mlflow.log_artifact' call?

Accepted Answer

Git commit hash and data URI. Capturing the specific code version (git hash) and data snapshot ID ensures reproducibility.

Answer

Environment variable list

Answer

User IP address

Answer

Compute cluster name

Question 5

In Google Vertex AI Pipelines, which mechanism prevents 'training-serving skew' when deploying a custom container model?

Accepted Answer

Model Monitoring skew detection. Vertex AI Model Monitoring provides built-in skew detection by comparing training data statistics against serving traffic statistics.

Answer

Custom prediction routines

Answer

Artifact metadata locks

Answer

Pipeline concurrency limits

Question 6

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

Accepted Answer

Feature Attribution Drift (SHAP). Feature attribution drift monitors how the model's reliance on specific input features changes over time.

Answer

Cosine Similarity

Answer

Mean Square Error

Answer

Precision-Recall AUC

Question 7

Which AWS SageMaker feature allows auditors to verify that a model was trained on encrypted data?

Accepted Answer

KMS Key Usage in CloudTrail. SageMaker Training Jobs support 'Data Encryption' via KMS keys, which is logged in CloudTrail.

Answer

Instance type selection

Answer

Model registry labels

Answer

Notebook instance tags

Question 8

In Azure Machine Learning, which component should an auditor verify to ensure model registry access is restricted to authorized CI/CD service principals?

Accepted Answer

Workspace Access Control (IAM). Access to the Model Registry is governed by Azure RBAC and specifically Managed Identities/Service Principals assigned to the workspace.

Answer

Environment YAML files

Answer

Compute Instance logs

Answer

Data labeling projects

Question 9

What is the primary function of an 'AI Pipeline Orchestrator' like Kubeflow or Airflow?

Accepted Answer

Automating task dependencies. Orchestration automates the sequence of tasks: data ingestion, training, and deployment.

Answer

Encrypting network traffic

Answer

Writing code for the model

Answer

Optimizing SQL queries

Question 10

In the context of AI Operations, what does 'CI/CD' stand for?

Accepted Answer

Continuous Integration and Continuous Deployment. Continuous Integration and Continuous Deployment/Delivery.

Answer

Computer Intelligence and Cloud Data

Answer

Continuous Inference and Cloud Deployment

Answer

Code Integration and Cloud Delivery

Question 11

Which TWO of the following are essential for ensuring 'Model Observability' in an enterprise production environment?

Accepted Answer

System latency monitoring. Observability requires both performance metrics and input monitoring (drift).

Answer

Manual code refactoring

Answer

Database backup frequency

Answer

Developer salary audits

Question 12

Which THREE items must be included in a 'Model Card' to comply with regulatory model risk management standards?

Accepted Answer

Intended use cases. Model cards summarize intended use, limitations, and performance characteristics.

Answer

Internal employee passwords

Answer

Cloud service provider revenue

Question 13

In a CI/CD pipeline for AI, where should the 'Model Validation' gate be located?

Accepted Answer

Between training and registration. Validation must occur after training but before model registration to prevent bad models from reaching production.

Answer

During feature engineering

Answer

Before source code check-in

Answer

After deployment

Question 14

When using DVC (Data Version Control) in an AI project, what is the best practice to ensure the 'data version' is immutable for audit purposes?

Accepted Answer

Commit .dvc files to git. Committing the '.dvc' file (which contains a hash of the data) to Git ensures that the specific data version is locked to the code version.

Answer

Use absolute file paths

Answer

Disable data caching

Answer

Store data on local drive

Question 15

During an audit of a model inference endpoint, you observe that 'Model Explanation' logs are missing. Which library or tool was likely omitted from the deployment?

Accepted Answer

SHAP integration. Tools like Captum (PyTorch), SHAP, or SageMaker Clarify are required to generate inference-time explanations.

Answer

Kubernetes manifest

Answer

TensorBoard logging

Answer

Prometheus exporter