AAIA AI Operations — Timed 10 Questions

Question 1

When auditing Kubeflow pipelines, which configuration file is used to define the resource requests for individual pipeline steps?

Accepted Answer

resource_spec in the SDK. In Kubeflow Pipelines, resource requests and limits for individual pipeline steps are defined in the SDK using the resource_spec parameter (e.g., set_cpu_limit, set_memory_limit, set_gpu_limit) on the ContainerOp or via the @dsl.component decorator. This specification is compiled into the Argo Workflow YAML that Kubernetes uses to schedule the pod. It is the correct, step-level configuration mechanism.

Answer

Namespace policy

Answer

Kubeconfig file

Answer

Service mesh config

Question 2

An organization uses MLflow for model tracking. To ensure auditability of model lineage, what must be captured in the 'mlflow.log_artifact' call?

Accepted Answer

Git commit hash and data URI. Capturing the specific code version (git hash) and data snapshot ID ensures reproducibility.

Answer

Environment variable list

Answer

User IP address

Answer

Compute cluster name

Question 3

In Google Vertex AI Pipelines, which mechanism prevents 'training-serving skew' when deploying a custom container model?

Accepted Answer

Model Monitoring skew detection. Model Monitoring skew detection in Vertex AI Pipelines continuously compares the statistical distribution of live prediction requests against the training data distribution, alerting when drift or skew exceeds configured thresholds. This directly addresses training-serving skew by detecting when the model's input features in production diverge from those seen during training. It is the only option that provides an automated, ongoing mechanism specifically designed to catch this mismatch.

Answer

Custom prediction routines

Answer

Artifact metadata locks

Answer

Pipeline concurrency limits

Question 4

What is the primary purpose of a 'Champion-Challenger' deployment pattern in MLOps?

Accepted Answer

To safely test a new model vs production. Champion-Challenger deployment runs the current production model (champion) alongside one or more candidate models (challengers), routing a small percentage of live traffic to the challengers and comparing business/ML metrics. This enables safe, data-driven promotion decisions without risking the full user base.

Answer

To manage database indexes

Answer

To compress model weights

Answer

To increase training speed

Question 5

For a production LLM deployment, which operational metric is most critical to monitor to identify 'hallucination' or grounding failure in real-time?

Accepted Answer

Context faithfulness score. Retrieval metrics, specifically faithfulness or answer relevance, indicate if the model is hallucinating outside of retrieved context.

Answer

Throughput (TPS)

Answer

Token generation latency

Answer

Error code distribution

Answer

Memory footprint

Question 6

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

Accepted Answer

Feature Attribution Drift (SHAP). Feature Attribution Drift, computed using SHAP (SHapley Additive exPlanations) values, measures how the contribution of each input feature to the model's prediction changes over time relative to a baseline. This is the correct metric for detecting feature attribution shift in structured data within SageMaker Model Monitor. It captures not just input distribution changes but changes in how the model weighs features.

Answer

Cosine Similarity

Answer

Mean Square Error

Answer

Precision-Recall AUC

Question 7

Which AWS SageMaker feature allows auditors to verify that a model was trained on encrypted data?

Accepted Answer

KMS Key Usage in CloudTrail. SageMaker Training Jobs support 'Data Encryption' via KMS keys, which is logged in CloudTrail.

Answer

Instance type selection

Answer

Model registry labels

Answer

Notebook instance tags

Question 8

In Azure Machine Learning, which component should an auditor verify to ensure model registry access is restricted to authorized CI/CD service principals?

Accepted Answer

Workspace Access Control (IAM). Access to the Model Registry is governed by Azure RBAC and specifically Managed Identities/Service Principals assigned to the workspace.

Answer

Environment YAML files

Answer

Compute Instance logs

Answer

Data labeling projects

Question 9

What is the primary function of an 'AI Pipeline Orchestrator' like Kubeflow or Airflow?

Accepted Answer

Automating task dependencies. Orchestration automates the sequence of tasks: data ingestion, training, and deployment.

Answer

Encrypting network traffic

Answer

Writing code for the model

Answer

Optimizing SQL queries

Question 10

In the context of AI Operations, what does 'CI/CD' stand for?

Accepted Answer

Continuous Integration and Continuous Deployment. Continuous Integration and Continuous Deployment/Delivery.

Answer

Computer Intelligence and Cloud Data

Answer

Continuous Inference and Cloud Deployment

Answer

Code Integration and Cloud Delivery