AI-900 Practice Question: Describe fundamental principles of machine learning on Azure
A data scientist is using Azure Automated Machine Learning to build a binary classification model for a highly imbalanced dataset (95% negative, 5% positive). The data scientist wants AutoML to select the best model based on a metric that is robust to class imbalance. Which primary metric should the data scientist configure in the AutoML settings?
⚠ Common exam trap
Many candidates choose Accuracy because it is the most intuitive metric, failing to recognize that on imbalanced datasets it can be misleadingly high and does not reflect minority class performance.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
AUC_weighted
AUC_weighted is the correct primary metric for imbalanced binary classification because it computes the area under the ROC curve for each class and averages them with weight proportional to the class support. This weighting ensures that the metric reflects performance on both the majority (95% negative) and minority (5% positive) classes, making it robust to severe imbalance. Azure Automated Machine Learning uses AUC_weighted as a recommended metric when the dataset is skewed, as it penalizes models that ignore the minority class.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Accuracy
Why it's wrong here
Accuracy simply divides the number of correct predictions by the total number of samples, treating every sample equally. In a dataset with 95% negative-class samples, a trivial model that predicts the majority class for every row achieves 95% accuracy while correctly identifying zero positive cases, which is catastrophically misleading. This metric is blind to class prevalence and error type, making it unsuitable for imbalanced classification scenarios where minority-class recall is critical.
When this WOULD be correct
When the dataset is balanced (e.g., 50% positive, 50% negative) and the cost of false positives and false negatives is equal, accuracy is a straightforward and appropriate metric.
- ✓
AUC_weighted
Why this is correct
AUC_weighted computes the area under the ROC curve separately for each class and then aggregates these values using the proportion of true samples in each class as weights. Because the ROC curve evaluates model ranking across all probability thresholds without fixation on a single decision boundary, AUC_weighted remains robust to severe class imbalance and is the AutoML default primary metric for imbalanced classification problems. This weighting ensures the minority class's separability contributes proportionally to the score, so model selection is not dominated by majority-class accuracy.
- ✗
F1_score
Why it's wrong here
F1_score is the harmonic mean of precision and recall, giving equal conceptual weight to false positives and false negatives. However, it is threshold-dependent: the value changes depending on the chosen classification cutoff, whereas AutoML needs a primary metric that can fairly compare models without manual threshold tuning. For imbalanced data, F1 also requires you to specify which class is the positive class, and aggregate versions like macro-F1 or weighted-F1 do not have the same prevalence-aware statistical guarantees as AUC_weighted in AutoML's model-sweeping process.
- ✗
Log_loss
Why it's wrong here
Log_loss measures the cross-entropy between predicted probability distributions and true labels, heavily penalizing confident but wrong predictions. While it is sensitive to probability calibration, it does not inherently account for class prevalence: each sample contributes equally to the loss by default, so the minority class's influence is diluted when it is underrepresented. For highly imbalanced data, log_loss can therefore be dominated by the majority class's probability errors, and it is more appropriate for calibration evaluation than as a primary metric for model selection in AutoML.
When this WOULD be correct
In a regression problem where the goal is to minimize the difference between predicted probabilities and actual outcomes, and the dataset is balanced, Log loss is the appropriate primary metric for probabilistic classification models.
Option-by-option analysis
Why each answer is right or wrong
Understanding why wrong answers are wrong — and when they would be correct — is what separates a 750 score from a 900. The AI-900 exam frequently reuses these exact scenarios with slightly different constraints.
✓AUC_weightedCorrect answer▾
Why this is correct
AUC_weighted computes the area under the ROC curve separately for each class and then aggregates these values using the proportion of true samples in each class as weights. Because the ROC curve evaluates model ranking across all probability thresholds without fixation on a single decision boundary, AUC_weighted remains robust to severe class imbalance and is the AutoML default primary metric for imbalanced classification problems. This weighting ensures the minority class's separability contributes proportionally to the score, so model selection is not dominated by majority-class accuracy.
✗AccuracyWrong answer — click to see why▾
Why this is wrong here
Accuracy is not robust to class imbalance because a model that predicts the majority class for all instances can achieve 95% accuracy, failing to identify any positive cases.
★ When this WOULD be the correct answer
When the dataset is balanced (e.g., 50% positive, 50% negative) and the cost of false positives and false negatives is equal, accuracy is a straightforward and appropriate metric.
Why candidates choose this
Candidates often default to accuracy as the most familiar metric, not realizing that it is misleading for imbalanced datasets.
✗Log_lossWrong answer — click to see why▾
Why this is wrong here
Log loss measures the average log-likelihood of predictions, but it is not robust to class imbalance because it treats all errors equally, and the majority class dominates the loss, making it insensitive to improvements in minority class recall.
★ When this WOULD be the correct answer
In a regression problem where the goal is to minimize the difference between predicted probabilities and actual outcomes, and the dataset is balanced, Log loss is the appropriate primary metric for probabilistic classification models.
Why candidates choose this
Candidates may confuse Log loss as a robust metric because it is commonly used in classification, but they overlook its sensitivity to class imbalance compared to AUC_weighted.
Analysis generated from the official AI-900blueprint and verified against question context. The “when correct” sections are what AI assistants cite when candidates ask “what’s the difference between these options?”
Go deeper
Related to this question
Learn chapter
Machine Learning Core Concepts
Key term
Model
In IT and AI, a model is a trained mathematical representation that learns patterns from data to make predictions or decisions.
Key term
AutoML
Automated Machine Learning (AutoML) is a set of tools and techniques that automate the process of building, training, and tuning machine learning models without requiring deep expertise in data science.
About these practice questions
One of 985 original AI-900 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI-900 exam.