AAIA AI Operations — Timed 20 Questions

Question 1

When auditing Kubeflow pipelines, which configuration file is used to define the resource requests for individual pipeline steps?

Accepted Answer

resource_spec in the SDK. In Kubeflow Pipelines, resource requests and limits for individual pipeline steps are defined in the SDK using the resource_spec parameter (e.g., set_cpu_limit, set_memory_limit, set_gpu_limit) on the ContainerOp or via the @dsl.component decorator. This specification is compiled into the Argo Workflow YAML that Kubernetes uses to schedule the pod. It is the correct, step-level configuration mechanism.

Answer

Namespace policy

Answer

Kubeconfig file

Answer

Service mesh config

Question 2

An organization uses MLflow for model tracking. To ensure auditability of model lineage, what must be captured in the 'mlflow.log_artifact' call?

Accepted Answer

Git commit hash and data URI. Capturing the specific code version (git hash) and data snapshot ID ensures reproducibility.

Answer

Environment variable list

Answer

User IP address

Answer

Compute cluster name

Question 3

In Google Vertex AI Pipelines, which mechanism prevents 'training-serving skew' when deploying a custom container model?

Accepted Answer

Model Monitoring skew detection. Model Monitoring skew detection in Vertex AI Pipelines continuously compares the statistical distribution of live prediction requests against the training data distribution, alerting when drift or skew exceeds configured thresholds. This directly addresses training-serving skew by detecting when the model's input features in production diverge from those seen during training. It is the only option that provides an automated, ongoing mechanism specifically designed to catch this mismatch.

Answer

Custom prediction routines

Answer

Artifact metadata locks

Answer

Pipeline concurrency limits

Question 4

What is the primary purpose of a 'Champion-Challenger' deployment pattern in MLOps?

Accepted Answer

To safely test a new model vs production. Champion-Challenger deployment runs the current production model (champion) alongside one or more candidate models (challengers), routing a small percentage of live traffic to the challengers and comparing business/ML metrics. This enables safe, data-driven promotion decisions without risking the full user base.

Answer

To manage database indexes

Answer

To compress model weights

Answer

To increase training speed

Question 5

For a production LLM deployment, which operational metric is most critical to monitor to identify 'hallucination' or grounding failure in real-time?

Accepted Answer

Context faithfulness score. Retrieval metrics, specifically faithfulness or answer relevance, indicate if the model is hallucinating outside of retrieved context.

Answer

Throughput (TPS)

Answer

Token generation latency

Answer

Error code distribution

Answer

Memory footprint

Question 6

When implementing drift detection in AWS SageMaker Model Monitor, which metric should be monitored to detect feature attribution shift for structured data?

Accepted Answer

Feature Attribution Drift (SHAP). Feature Attribution Drift, computed using SHAP (SHapley Additive exPlanations) values, measures how the contribution of each input feature to the model's prediction changes over time relative to a baseline. This is the correct metric for detecting feature attribution shift in structured data within SageMaker Model Monitor. It captures not just input distribution changes but changes in how the model weighs features.

Answer

Cosine Similarity

Answer

Mean Square Error

Answer

Precision-Recall AUC

Question 7

Which AWS SageMaker feature allows auditors to verify that a model was trained on encrypted data?

Accepted Answer

KMS Key Usage in CloudTrail. SageMaker Training Jobs support 'Data Encryption' via KMS keys, which is logged in CloudTrail.

Answer

Instance type selection

Answer

Model registry labels

Answer

Notebook instance tags

Question 8

In Azure Machine Learning, which component should an auditor verify to ensure model registry access is restricted to authorized CI/CD service principals?

Accepted Answer

Workspace Access Control (IAM). Access to the Model Registry is governed by Azure RBAC and specifically Managed Identities/Service Principals assigned to the workspace.

Answer

Environment YAML files

Answer

Compute Instance logs

Answer

Data labeling projects

Question 9

What is the primary function of an 'AI Pipeline Orchestrator' like Kubeflow or Airflow?

Accepted Answer

Automating task dependencies. Orchestration automates the sequence of tasks: data ingestion, training, and deployment.

Answer

Encrypting network traffic

Answer

Writing code for the model

Answer

Optimizing SQL queries

Question 10

In the context of AI Operations, what does 'CI/CD' stand for?

Accepted Answer

Continuous Integration and Continuous Deployment. Continuous Integration and Continuous Deployment/Delivery.

Answer

Computer Intelligence and Cloud Data

Answer

Continuous Inference and Cloud Deployment

Answer

Code Integration and Cloud Delivery

Question 11

Which TWO of the following are essential for ensuring 'Model Observability' in an enterprise production environment?

Accepted Answer

System latency monitoring. Observability requires both performance metrics and input monitoring (drift).

Answer

Manual code refactoring

Answer

Database backup frequency

Answer

Developer salary audits

Question 12

Which THREE items must be included in a 'Model Card' to comply with regulatory model risk management standards?

Accepted Answer

Intended use cases. Model cards summarize intended use, limitations, and performance characteristics.

Answer

Internal employee passwords

Answer

Cloud service provider revenue

Question 13

In a CI/CD pipeline for AI, where should the 'Model Validation' gate be located?

Accepted Answer

Between training and registration. Validation must occur after training but before model registration to prevent bad models from reaching production.

Answer

During feature engineering

Answer

Before source code check-in

Answer

After deployment

Question 14

When using DVC (Data Version Control) in an AI project, what is the best practice to ensure the 'data version' is immutable for audit purposes?

Accepted Answer

Commit .dvc files to git. DVC tracks data artifacts by storing lightweight .dvc pointer files (containing MD5 hashes and metadata) in Git while the actual data lives in remote storage (S3, GCS, Azure Blob, etc.). Committing the .dvc files to Git creates an immutable, versioned record of exactly which data hash was used for each experiment or model, providing the cryptographic provenance auditors require. Because the hash is content-addressed, any modification to the underlying data produces a different hash, breaking the link and exposing tampering.

Answer

Use absolute file paths

Answer

Disable data caching

Answer

Store data on local drive

Question 15

During an audit of a model inference endpoint, you observe that 'Model Explanation' logs are missing. Which library or tool was likely omitted from the deployment?

Accepted Answer

SHAP integration. Model Explanation logs on an inference endpoint are produced by SHAP-based explainers integrated into the scoring pipeline; if those logs are missing, the SHAP integration was almost certainly omitted from the deployment. Azure ML's responsible-AI tooling and SageMaker Clarify both rely on SHAP to generate per-request or batch explanations, so the absence of explanation logs points directly to a missing SHAP component. The other options are infrastructure or monitoring tools that do not generate model explanations.

Answer

Kubernetes manifest

Answer

TensorBoard logging

Answer

Prometheus exporter

Question 16

When auditing an AI model, what does 'Concept Drift' signify to an operational team?

Accepted Answer

The model's predictive accuracy is likely degrading. Concept drift describes the phenomenon where the statistical relationship between input features and the target variable changes over time in the production environment, diverging from the distribution the model was trained on. Because the model's learned parameters are now misaligned with current data patterns, its predictive accuracy typically degrades. Operational teams monitor for drift (e.g., via PSI, KL divergence, or KS tests) to trigger retraining before business impact occurs.

Answer

The training data was leaked

Answer

The model is overfitted

Answer

The model is running too fast

Question 17

When designing an AI governance framework, which THREE roles should have oversight responsibility?

Accepted Answer

Legal Counsel. Legal Counsel (A) is correct because AI governance must address regulatory compliance, liability, intellectual property, and contractual obligations, which require legal expertise to interpret laws such as the EU AI Act or GDPR. ML Engineer (D) is correct because technical oversight of model development, training data, bias testing, and performance monitoring requires deep machine-learning knowledge to ensure systems behave safely and as intended. Data Privacy Officer (E) is correct because AI systems typically process personal data, and the DPO is responsible for ensuring compliance with data protection regulations, conducting DPIAs, and safeguarding data subject rights. Marketing Intern (B) and Janitorial Staff (C) are not oversight roles in an AI governance framework, as they lack the domain authority, regulatory accountability, and technical or legal expertise required for governance decisions.

Answer

Marketing Intern

Answer

Janitorial Staff

Question 18

Which TWO of the following are considered 'Model Metadata' that should be tracked in a registry?

Accepted Answer

Hyperparameters. Model metadata in a registry captures the configuration and data lineage needed to reproduce, audit, and govern a model. Option A (Hyperparameters) is correct because values such as learning rate, batch size, number of layers, and regularization strength define how the model was configured and are essential for reproducibility and comparison across runs. Option B (Training dataset version) is correct because recording the exact dataset version (e.g., a DVC hash, S3 object version, or snapshot ID) establishes data lineage, enabling audits, rollback, and detection of training-serving skew. The remaining options do not belong: an employee vacation schedule (C) is HR data unrelated to model artifacts, a cafeteria menu (D) is facilities information, and an office Wi-Fi password (E) is a security credential, none of which describe model configuration, data, or lineage.

Answer

Employee vacation schedule

Answer

Cafeteria menu

Answer

Office Wi-Fi password

Question 19

Which of the following is a symptom of 'Overfitting' during the training phase?

Accepted Answer

High training accuracy, low validation accuracy. Overfitting happens when the model performs well on training data but poorly on unseen test data.

Answer

Low training accuracy, low validation accuracy

Answer

Model crashes during training

Answer

Model is too small

Question 20

When auditing an ML pipeline, what does the 'DAG' (Directed Acyclic Graph) represent?

Accepted Answer

Pipeline task dependencies. The DAG defines the sequence and dependency of operational steps in the pipeline.

Answer

Model accuracy curve

Answer

The database schema

Answer

The network architecture