Troubleshooting Scenario Questions

The intern's role is not included in the allowed roles

Why wrong: That alone wouldn't cause denial if not explicitly denied.

The policy JSON has a syntax error

Why wrong: The JSON is valid.

The endpoint path is incorrect

Why wrong: Exhibit shows correct path.

Increase the number of hidden layers

Why wrong: More layers increase complexity and may worsen oscillations.

Decrease the batch size

Why wrong: Smaller batch sizes introduce more noise, which can increase oscillations.

Use a test set

Why wrong: Test set is for evaluation, not for training stability.

Transfer learning

Why wrong: Transfer learning is a technique, not a problem description.

Underfitting

Why wrong: Underfitting would show poor performance even on training data.

Bias-variance tradeoff

Why wrong: This describes the balance but not the specific failure to generalize.

Replace the convolutional layers with transformer layers to improve attention.

Why wrong: Changing architecture typically requires full retraining and may not address low-light specifically.

Use adversarial training to add imperceptible perturbations to training images.

Why wrong: Adversarial training improves resistance to malicious perturbations but not necessarily to natural low-light conditions.

Increase the model's depth by adding more convolutional layers.

Why wrong: Increasing depth changes architecture and requires retraining; it may not directly address low-light performance.

Automatically retrain the model with a balanced dataset including more pediatric cases.

Why wrong: Retraining without confirming the cause may not solve the problem and could introduce other issues.

Expand the test set with more pediatric X-rays and re-evaluate overall accuracy.

Why wrong: Overall accuracy may still hide subgroup disparities; need stratified analysis.

Add more features to the model to capture age-related anatomical differences.

Why wrong: Feature engineering might help but is not the first diagnostic step.

Replace the model with a third-party vendor model that claims to be bias-free.

Why wrong: Switching vendors does not ensure fairness and may introduce new issues; audit first.

Re-sample the training data to have equal numbers of applicants over and under 60.

Why wrong: Re-sampling without bias assessment may not address the root cause and could hurt model performance.

Remove the age feature from the training data and retrain the model.

Why wrong: Other features may correlate with age, so removal alone may not eliminate bias; also lacks measurement.

Precision

Why wrong: Precision focuses on false positives, but the main issue is missing actual frauds.

F1-score

Why wrong: F1-score balances precision and recall, but recall is the primary concern.

AUC-ROC

Why wrong: AUC-ROC is a general measure but does not directly address the failure to detect frauds.

Mean Squared Error; use L2 regularization

Why wrong: MSE is for regression; regularization does not address class imbalance.

F1 score; use principal component analysis

Why wrong: F1 is good but PCA reduces features, not imbalance; SMOTE is more direct.

Accuracy; collect more data

Why wrong: Accuracy is misleading in imbalance; more data of majority class may not help.

Precision

Why wrong: Precision measures accuracy of positive predictions, but the priority is catching actual churners.

F1-score

Why wrong: F1-score is a harmonic mean of precision and recall, but recall alone is more appropriate here.

Specificity

Why wrong: Specificity measures true negative rate, not relevant for identifying churners.

Underfitting; increase the number of time steps

Why wrong: Increasing time steps can worsen vanishing gradients and does not address the core issue.

Exploding gradients; apply gradient clipping

Why wrong: Exploding gradients cause large updates, but do not specifically hinder long-range dependencies; vanishing gradients are the typical culprit.

Overfitting; reduce the number of layers

Why wrong: Overfitting is not indicated; the model fails to learn, not memorize.

Apply matrix factorization with higher latent factors

Why wrong: Matrix factorization still depends on interaction data and may not capture niche item features.

Increase the weight of popular items in the recommendation score

Why wrong: This would further bias toward popular items and worsen the long-tail problem.

Collect more user interaction data over time

Why wrong: More data won't help if niche items inherently have few interactions.

F1 score

Why wrong: F1 balances precision and recall, but recall alone is more appropriate when missing fraud is costly.

Precision

Why wrong: Precision focuses on false positives, not on catching fraud.

Accuracy

Why wrong: Accuracy is misleading on imbalanced datasets.

Increase the number of layers in the network.

Why wrong: Adding layers does not address catastrophic forgetting; it may even introduce more parameters that can overwrite previous knowledge.

Train the model from scratch using only real-world data.

Why wrong: This discards all simulation knowledge, which is wasteful and may lead to poor performance due to limited real data.

Increase the learning rate during fine-tuning.

Why wrong: A higher learning rate would cause larger weight updates, accelerating forgetting.

Implement a monitoring dashboard to track drift over time and set up alerts.

Why wrong: Monitoring is proactive but does not recover the immediate performance drop; it should be done after resolving the issue.

Use feature selection to remove the drifting features and retrain.

Why wrong: Removing drifting features may discard important information; the drift might be due to a pipeline error that can be fixed.

Immediately retrain the model on all available data including new production data.

Why wrong: Retraining on drifted data without understanding the cause may encode faulty patterns and not fix the underlying issue.

Document the model's predictions and submit a report to the regulator explaining the historical dataset bias

Why wrong: Documentation is important but does not remediate the bias; the regulator expects corrective action.

Immediately deploy a rule-based system to manually review all denial decisions from the AI system

Why wrong: This is a temporary measure but does not solve the bias in the model itself.

Disclose the bias findings to all rejected applicants and offer them priority reconsideration

Why wrong: Transparency is good, but without actually fixing the model, the same bias will continue.