Operational Procedures practice questions

An organization wants to implement an AI ethics board. Which composition best ensures independence and expertise?

Trap 1: All members from the legal department

Legal focuses on compliance, not broader ethics.

Trap 2: IT department head and data scientists

Lacks ethical training and independence.

Trap 3: Only senior executives from the company

Executives may have conflicts of interest and lack ethics expertise.

A
All members from the legal department
Why wrong: Legal focuses on compliance, not broader ethics.
B
IT department head and data scientists
Why wrong: Lacks ethical training and independence.
C
Mix of internal stakeholders and external ethicists
Ensures independence and diverse expertise.
D
Only senior executives from the company
Why wrong: Executives may have conflicts of interest and lack ethics expertise.

Question 2easymultiple choice

What is the primary function of an AI ethics board within an organization?

Trap 1: Developing algorithms

Algorithm development is typically done by data scientists, not the ethics board.

Trap 2: Managing cloud infrastructure

Cloud infrastructure is an IT operations function.

Trap 3: Marketing AI products

Marketing is separate from ethics governance.

A
Developing algorithms
Why wrong: Algorithm development is typically done by data scientists, not the ethics board.
B
Managing cloud infrastructure
Why wrong: Cloud infrastructure is an IT operations function.
C
Marketing AI products
Why wrong: Marketing is separate from ethics governance.
D
Reviewing AI projects for ethical compliance
The board provides oversight and guidance on ethical matters.

Question 3mediummulti select

An AI team is deploying a real-time document intelligence service that extracts key-value pairs from invoices. The pipeline includes an LLM that calls a function to parse structured output. Which TWO testing strategies are essential before production deployment?

Trap 1: Load testing to simulate peak invoice volume (e.g., end of month)

Important for performance but not an essential pre-deployment test for correctness.

Trap 2: Regression tests on the model training pipeline to ensure the base…

If using a hosted LLM, the model may change; but testing the training pipeline is not relevant for a pre-trained model.

Trap 3: Unit tests for the data pipeline that cleans and normalises invoice…

Important but not the most essential for the LLM function calling aspect.

A
An evaluation framework that compares extracted fields against ground truth for a test set of invoices
Measures accuracy (e.g., precision, recall) of the extraction, critical for business requirements.
B
Load testing to simulate peak invoice volume (e.g., end of month)
Why wrong: Important for performance but not an essential pre-deployment test for correctness.
C
Regression tests on the model training pipeline to ensure the base LLM hasn't changed
Why wrong: If using a hosted LLM, the model may change; but testing the training pipeline is not relevant for a pre-trained model.
D
Integration tests that call the LLM API with sample invoices and verify the JSON output structure
Ensures the API, function definitions, and parsing work correctly.
E
Unit tests for the data pipeline that cleans and normalises invoice images
Why wrong: Important but not the most essential for the LLM function calling aspect.

Question 4easymultiple choice

An organisation is developing an AI policy. According to the NIST AI RMF, which function involves establishing policies and procedures to ensure the organisation governs AI responsibly?

Trap 1: Manage

Manage deals with risk treatment and response.

Trap 2: Measure

Measure involves quantitative metrics for risk and trustworthiness.

Trap 3: Map

Map focuses on understanding the AI system's context and risks.

A
Manage
Why wrong: Manage deals with risk treatment and response.
B
Measure
Why wrong: Measure involves quantitative metrics for risk and trustworthiness.
C
Govern
Govern involves setting policies, roles, and responsibilities for AI governance.
D
Map
Why wrong: Map focuses on understanding the AI system's context and risks.

Question 5hardmultiple choice

A large hospital system deploys an AI triage system for emergency rooms. The system uses patient vitals and symptoms to recommend treatment priority. Six months after deployment, complaints arise that the system frequently underestimates the severity of symptoms for patients from certain ethnic backgrounds. A data scientist runs a bias audit and finds that the model's false negative rate is 20% higher for the minority group. The hospital's AI governance board requires immediate corrective action. The data science team has limited resources and cannot retrain the entire model from scratch. They have access to the training data, which is imbalanced. The model is a gradient boosted tree. Which course of action best addresses the bias while minimizing operational impact?

Trap 1: Rebalance the training data using SMOTE and retrain the model

SMOTE retraining requires time and may not fully resolve bias if the model still learns proxy features.

Trap 2: Use adversarial debiasing during training to remove protected…

Adversarial debiasing requires retraining, which contradicts the limited resources constraint.

Trap 3: Replace the model with a simpler logistic regression model to…

Switching models is a major change and may degrade overall performance.

A
Rebalance the training data using SMOTE and retrain the model
Why wrong: SMOTE retraining requires time and may not fully resolve bias if the model still learns proxy features.
B
Use adversarial debiasing during training to remove protected attribute correlations
Why wrong: Adversarial debiasing requires retraining, which contradicts the limited resources constraint.
C
Post-process the model's predictions by adjusting thresholds for the minority group
Threshold adjustment is fast, cheap, and directly minimizes false negative disparity.
D
Replace the model with a simpler logistic regression model to improve interpretability
Why wrong: Switching models is a major change and may degrade overall performance.

Question 6hardmultiple choice

A global retailer uses an AI model to forecast demand across thousands of stores. After deployment, the model's predictions become less accurate during holiday seasons. The training data included two years of holiday periods. What is the most effective operational strategy to handle this recurring seasonal drift?

Trap 1: Deploy an anomaly detection system to flag holiday prediction…

This is reactive, not proactive.

Trap 2: Use an ensemble of models trained on different time periods

Ensembles can help but not specifically target seasonal retraining.

Trap 3: Increase the volume of training data by including five years of…

More data may not capture the specific seasonal shift adequately.

A
Deploy an anomaly detection system to flag holiday prediction outliers
Why wrong: This is reactive, not proactive.
B
Implement a scheduled retraining cycle just before each holiday period
Proactive retraining with recent holiday data mitigates seasonal drift.
C
Use an ensemble of models trained on different time periods
Why wrong: Ensembles can help but not specifically target seasonal retraining.
D
Increase the volume of training data by including five years of history
Why wrong: More data may not capture the specific seasonal shift adequately.

Question 7mediummulti select

A company is implementing an AI ethics board. Which TWO responsibilities should the board typically have?

Trap 1: Writing code for fairness algorithms

Writing code is a technical implementation task, not a governance board responsibility.

Trap 2: Developing marketing strategies for AI products

Marketing strategies are outside the scope of an ethics board's governance role.

Trap 3: Conducting daily data privacy audits

Daily audits are operational tasks handled by privacy or compliance teams, not the ethics board.

A
Approving or rejecting high-risk AI initiatives
The board should have authority to approve or reject high-risk initiatives to ensure responsible AI deployment.
B
Writing code for fairness algorithms
Why wrong: Writing code is a technical implementation task, not a governance board responsibility.
C
Reviewing AI projects for ethical compliance and potential biases
The board should review projects to ensure they adhere to ethical guidelines and identify biases.
D
Developing marketing strategies for AI products
Why wrong: Marketing strategies are outside the scope of an ethics board's governance role.
E
Conducting daily data privacy audits
Why wrong: Daily audits are operational tasks handled by privacy or compliance teams, not the ethics board.

Question 8easymulti select

Which THREE are common pitfalls when operationalizing AI models? (Select THREE.)

Trap 1: Using simpler models that are easier to debug

Simpler models are often preferred for operationalization.

Trap 2: Automating the model retraining process

Automation is a best practice, not a pitfall.

A
Training-serving skew due to differences in data preprocessing
Causes model inaccuracies in production.
B
Using simpler models that are easier to debug
Why wrong: Simpler models are often preferred for operationalization.
C
Lack of monitoring for model performance drift
Without monitoring, degradation goes unnoticed.
D
Ignoring infrastructure scalability requirements
Leads to resource contention and failures.
E
Automating the model retraining process
Why wrong: Automation is a best practice, not a pitfall.