AAIA AI Operations Practice Test 2 — 15 Questions

Question 1

When auditing an AI model, what does 'Concept Drift' signify to an operational team?

Accepted Answer

The model's predictive accuracy is likely degrading. Concept drift describes the phenomenon where the statistical relationship between input features and the target variable changes over time in the production environment, diverging from the distribution the model was trained on. Because the model's learned parameters are now misaligned with current data patterns, its predictive accuracy typically degrades. Operational teams monitor for drift (e.g., via PSI, KL divergence, or KS tests) to trigger retraining before business impact occurs.

Answer

The training data was leaked

Answer

The model is overfitted

Answer

The model is running too fast

Question 2

When designing an AI governance framework, which THREE roles should have oversight responsibility?

Accepted Answer

Legal Counsel. Legal Counsel (A) is correct because AI governance must address regulatory compliance, liability, intellectual property, and contractual obligations, which require legal expertise to interpret laws such as the EU AI Act or GDPR. ML Engineer (D) is correct because technical oversight of model development, training data, bias testing, and performance monitoring requires deep machine-learning knowledge to ensure systems behave safely and as intended. Data Privacy Officer (E) is correct because AI systems typically process personal data, and the DPO is responsible for ensuring compliance with data protection regulations, conducting DPIAs, and safeguarding data subject rights. Marketing Intern (B) and Janitorial Staff (C) are not oversight roles in an AI governance framework, as they lack the domain authority, regulatory accountability, and technical or legal expertise required for governance decisions.

Answer

Marketing Intern

Answer

Janitorial Staff

Question 3

Which TWO of the following are considered 'Model Metadata' that should be tracked in a registry?

Accepted Answer

Hyperparameters. Model metadata in a registry captures the configuration and data lineage needed to reproduce, audit, and govern a model. Option A (Hyperparameters) is correct because values such as learning rate, batch size, number of layers, and regularization strength define how the model was configured and are essential for reproducibility and comparison across runs. Option B (Training dataset version) is correct because recording the exact dataset version (e.g., a DVC hash, S3 object version, or snapshot ID) establishes data lineage, enabling audits, rollback, and detection of training-serving skew. The remaining options do not belong: an employee vacation schedule (C) is HR data unrelated to model artifacts, a cafeteria menu (D) is facilities information, and an office Wi-Fi password (E) is a security credential, none of which describe model configuration, data, or lineage.

Answer

Employee vacation schedule

Answer

Cafeteria menu

Answer

Office Wi-Fi password

Question 4

Which of the following is a symptom of 'Overfitting' during the training phase?

Accepted Answer

High training accuracy, low validation accuracy. Overfitting happens when the model performs well on training data but poorly on unseen test data.

Answer

Low training accuracy, low validation accuracy

Answer

Model crashes during training

Answer

Model is too small

Question 5

When auditing an ML pipeline, what does the 'DAG' (Directed Acyclic Graph) represent?

Accepted Answer

Pipeline task dependencies. The DAG defines the sequence and dependency of operational steps in the pipeline.

Answer

Model accuracy curve

Answer

The database schema

Answer

The network architecture

Question 6

Which security control is most effective against 'Prompt Injection' attacks in a deployed LLM?

Accepted Answer

Prompt Guardrails. Prompt Guardrails are input/output filtering layers that detect and block malicious or policy-violating prompts, including injection attempts like 'ignore previous instructions' or role-override payloads. They work by classifying, pattern-matching, or using a secondary LLM to evaluate prompts before they reach the target model, and by sanitizing outputs before they reach the user. This is the most direct and effective control because it operates at the exact attack surface — the prompt itself.

Answer

Enabling GPU acceleration

Answer

Increasing model parameters

Answer

Regular re-training

Question 7

An auditor notices high 'latency' in an online model deployment. Which component should be inspected first?

Accepted Answer

Inference service resource limits. The inference container resource allocation or the request preprocessing overhead is the common bottleneck.

Answer

Developer laptop specs

Answer

Model documentation

Answer

Data labeling tool settings

Answer

Git repository history

Question 8

What is the function of a 'Feature Store' in an MLOps pipeline?

Accepted Answer

Centralized feature serving and storage. A Feature Store provides a centralized repository for standardized feature definitions and storage for both training and inference.

Answer

Storing raw video files

Answer

Compressing model weights

Answer

Monitoring compute costs

Question 9

If a model's performance drops, and the input data distribution has changed, what operational process must be triggered?

Accepted Answer

Model Retraining. Data distribution change triggers model retraining using updated data samples.

Answer

Security audit

Answer

Model Deletion

Answer

Hardware Upgrade

Answer

Code Refactoring

Question 10

What is the primary function of a 'Canary Deployment' in AI model release?

Accepted Answer

Gradual rollout to test stability. Releasing the new model to a small subset of traffic to monitor performance before full rollout.

Answer

To update documentation

Answer

To delete old models

Answer

To increase training performance

Answer

To encrypt model weights

Question 11

In SageMaker, what is the role of an 'Endpoint Configuration'?

Accepted Answer

Defines model variants and compute resources. Defines the model, instance type, and traffic distribution (variants) for the production endpoint.

Answer

Defines the database password

Answer

Defines the data cleaning logic

Answer

Defines the training budget

Question 12

Which environment should be used to perform 'Model Validation' before moving to production?

Accepted Answer

Staging Environment. Staging or UAT environments mirror production and are used for testing.

Answer

Training Cluster

Answer

Development Environment

Answer

Public Internet

Answer

Local Desktop

Question 13

What is the most effective way to audit 'Data Lineage' in a complex AI ecosystem?

Accepted Answer

Automated Metadata Tracking. Using metadata tracking tools that record the transformations from source to final model input.

Answer

Asking developers

Answer

Manual spreadsheet tracking

Answer

Checking project folder names

Question 14

Which TWO are common 'AI Security' threats that require operational oversight?

Accepted Answer

Model poisoning during training. Model poisoning during training (C) is a core AI security threat because an adversary who can influence the training data or pipeline can inject malicious samples that corrupt the learned model, degrading accuracy or implanting backdoors, so it demands operational oversight of data provenance, training integrity, and model validation. Adversarial evasion attacks (D) are also a common AI security threat: crafted inputs at inference time (e.g., perturbed images or prompts) cause the deployed model to misclassify or bypass controls, requiring ongoing monitoring, input filtering, and robustness testing in production. By contrast, slow training time (A) and high memory usage (B) are performance and resource-efficiency concerns, not security threats, and team burnout (E) is a personnel/wellbeing issue rather than an AI security risk.

Answer

Slow training time

Answer

High memory usage

Answer

Team burnout

Question 15

An organization experiences 'Data Leakage' in a machine learning project. What is the most likely cause during the training process?

Accepted Answer

Using target data in training features. Using test/validation data during feature selection or training leads to leakage.

Answer

Using standard Python libraries

Answer

Using secure cloud storage

Answer

Using too many GPUs