AAIA AI Operations Practice Test 1 — 15 Questions

Question 1

When auditing Kubeflow pipelines, which configuration file is used to define the resource requests for individual pipeline steps?

Accepted Answer

resource_spec in the SDK. In Kubeflow Pipelines, resource requests and limits for individual pipeline steps are defined in the SDK using the resource_spec parameter (e.g., set_cpu_limit, set_memory_limit, set_gpu_limit) on the ContainerOp or via the @dsl.component decorator. This specification is compiled into the Argo Workflow YAML that Kubernetes uses to schedule the pod. It is the correct, step-level configuration mechanism.

Answer

Namespace policy

Answer

Kubeconfig file

Answer

Service mesh config

Question 2

An organization uses MLflow for model tracking. To ensure auditability of model lineage, what must be captured in the 'mlflow.log_artifact' call?

Accepted Answer

Git commit hash and data URI. Capturing the specific code version (git hash) and data snapshot ID ensures reproducibility.

Answer

Environment variable list

Answer

User IP address

Answer

Compute cluster name

Question 3

In Google Vertex AI Pipelines, which mechanism prevents 'training-serving skew' when deploying a custom container model?

Accepted Answer

Model Monitoring skew detection. Model Monitoring skew detection in Vertex AI Pipelines continuously compares the statistical distribution of live prediction requests against the training data distribution, alerting when drift or skew exceeds configured thresholds. This directly addresses training-serving skew by detecting when the model's input features in production diverge from those seen during training. It is the only option that provides an automated, ongoing mechanism specifically designed to catch this mismatch.

Answer

Custom prediction routines

Answer

Artifact metadata locks

Answer

Pipeline concurrency limits

Question 4

What is the primary purpose of a 'Champion-Challenger' deployment pattern in MLOps?

Accepted Answer

To safely test a new model vs production. Champion-Challenger deployment runs the current production model (champion) alongside one or more candidate models (challengers), routing a small percentage of live traffic to the challengers and comparing business/ML metrics. This enables safe, data-driven promotion decisions without risking the full user base.

Answer

To manage database indexes

Answer

To compress model weights

Answer

To increase training speed

Question 5

For a production LLM deployment, which operational metric is most critical to monitor to identify 'hallucination' or grounding failure in real-time?

Accepted Answer

Context faithfulness score. Retrieval metrics, specifically faithfulness or answer relevance, indicate if the model is hallucinating outside of retrieved context.

Answer

Throughput (TPS)

Answer

Token generation latency

Answer

Error code distribution

Answer

Memory footprint

Question 6

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

Accepted Answer

Feature Attribution Drift (SHAP). Feature Attribution Drift, computed using SHAP (SHapley Additive exPlanations) values, measures how the contribution of each input feature to the model's prediction changes over time relative to a baseline. This is the correct metric for detecting feature attribution shift in structured data within SageMaker Model Monitor. It captures not just input distribution changes but changes in how the model weighs features.

Answer

Cosine Similarity

Answer

Mean Square Error

Answer

Precision-Recall AUC

Question 7

Which AWS SageMaker feature allows auditors to verify that a model was trained on encrypted data?

Accepted Answer

KMS Key Usage in CloudTrail. SageMaker Training Jobs support 'Data Encryption' via KMS keys, which is logged in CloudTrail.

Answer

Instance type selection

Answer

Model registry labels

Answer

Notebook instance tags

Question 8

In Azure Machine Learning, which component should an auditor verify to ensure model registry access is restricted to authorized CI/CD service principals?

Accepted Answer

Workspace Access Control (IAM). Access to the Model Registry is governed by Azure RBAC and specifically Managed Identities/Service Principals assigned to the workspace.

Answer

Environment YAML files

Answer

Compute Instance logs

Answer

Data labeling projects

Question 9

What is the primary function of an 'AI Pipeline Orchestrator' like Kubeflow or Airflow?

Accepted Answer

Automating task dependencies. Orchestration automates the sequence of tasks: data ingestion, training, and deployment.

Answer

Encrypting network traffic

Answer

Writing code for the model

Answer

Optimizing SQL queries

Question 10

In the context of AI Operations, what does 'CI/CD' stand for?

Accepted Answer

Continuous Integration and Continuous Deployment. Continuous Integration and Continuous Deployment/Delivery.

Answer

Computer Intelligence and Cloud Data

Answer

Continuous Inference and Cloud Deployment

Answer

Code Integration and Cloud Delivery

Question 11

Which TWO of the following are essential for ensuring 'Model Observability' in an enterprise production environment?

Accepted Answer

System latency monitoring. Observability requires both performance metrics and input monitoring (drift).

Answer

Manual code refactoring

Answer

Database backup frequency

Answer

Developer salary audits

Question 12

Which THREE items must be included in a 'Model Card' to comply with regulatory model risk management standards?

Accepted Answer

Intended use cases. Model cards summarize intended use, limitations, and performance characteristics.

Answer

Internal employee passwords

Answer

Cloud service provider revenue

Question 13

In a CI/CD pipeline for AI, where should the 'Model Validation' gate be located?

Accepted Answer

Between training and registration. Validation must occur after training but before model registration to prevent bad models from reaching production.

Answer

During feature engineering

Answer

Before source code check-in

Answer

After deployment

Question 14

When using DVC (Data Version Control) in an AI project, what is the best practice to ensure the 'data version' is immutable for audit purposes?

Accepted Answer

Commit .dvc files to git. DVC tracks data artifacts by storing lightweight .dvc pointer files (containing MD5 hashes and metadata) in Git while the actual data lives in remote storage (S3, GCS, Azure Blob, etc.). Committing the .dvc files to Git creates an immutable, versioned record of exactly which data hash was used for each experiment or model, providing the cryptographic provenance auditors require. Because the hash is content-addressed, any modification to the underlying data produces a different hash, breaking the link and exposing tampering.

Answer

Use absolute file paths

Answer

Disable data caching

Answer

Store data on local drive

Question 15

During an audit of a model inference endpoint, you observe that 'Model Explanation' logs are missing. Which library or tool was likely omitted from the deployment?

Accepted Answer

SHAP integration. Model Explanation logs on an inference endpoint are produced by SHAP-based explainers integrated into the scoring pipeline; if those logs are missing, the SHAP integration was almost certainly omitted from the deployment. Azure ML's responsible-AI tooling and SageMaker Clarify both rely on SHAP to generate per-request or batch explanations, so the absence of explanation logs points directly to a missing SHAP component. The other options are infrastructure or monitoring tools that do not generate model explanations.

Answer

Kubernetes manifest

Answer

TensorBoard logging

Answer

Prometheus exporter