AI Associate AI Fundamentals Practice Question
An admin creates a predictive model in Einstein Prediction Builder to forecast customer churn. The model shows high accuracy on test data but poor performance in production. What is the most likely cause?
⚠ Common exam trap
Salesforce often tests the distinction between overfitting (which affects test data) and data drift (which affects production data), trapping candidates who confuse high test accuracy with model generalization.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Data drift between training and production environments
The model's high accuracy on test data but poor performance in production is a classic symptom of data drift. In Einstein Prediction Builder, the model was trained on historical data that may not reflect current customer behavior patterns, leading to a mismatch between training and production distributions. This is not a model training issue but a data environment shift.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Improper feature scaling in the training pipeline
Why it's wrong here
Prediction Builder automatically handles scaling, so this is unlikely.
- ✗
The model is overfitted to the training data
Why it's wrong here
Overfitting would also cause low accuracy on test data, which is not indicated.
- ✗
Target leakage in the training dataset
Why it's wrong here
Target leakage would give artificially high accuracy in both test and production if leakage persists.
- ✓
Data drift between training and production environments
Why this is correct
Changes in customer behavior or data collection can make the model less effective.
Go deeper
Related to this question
About these practice questions
One of 753 original AI Associate practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on AI Associate
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. Refer to the exhibit. A Salesforce admin evaluates an Einstein Prediction Builder model for customer churn. What should be the admin's primary concern based on the exhibit?
easy- A.The model accuracy is too low for production use.
- ✓ B.The model has detected data drift, indicating the training data may no longer represent current patterns.
- C.The AUC is low, so the model is not better than random.
- D.The model uses too few features to be reliable.
Why B: The exhibit shows a data drift alert from Einstein Prediction Builder, which indicates that the statistical properties of the input data have changed over time. This is the admin's primary concern because a model trained on outdated patterns will produce unreliable predictions, even if its accuracy or AUC were initially high. Data drift directly undermines the model's validity in production.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This AI Associate practice question is part of Courseiva's free Salesforce certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the AI Associate exam.