You are a data scientist at a retail company. The company uses Einstein Discovery to analyze customer purchase patterns. The model is built on a dataset of 50,000 transactions. The model's R-squared is 0.85, but the predictions for new customers are consistently off by a large margin. The data includes features like 'Customer Age', 'Income', 'Previous Purchases', and 'Product Category'. The model was trained on data from the past two years. However, six months ago, the company launched a new loyalty program that significantly changed purchasing behavior. You suspect the model is not generalizing to new customers. What should you do to validate your hypothesis?
If performance is worse on recent data, concept drift is confirmed.
Why this answer
Creating a holdout set of transactions from the last six months directly tests whether the model's performance has degraded due to the loyalty program's impact on purchasing behavior. By comparing the R-squared or other metrics on this recent holdout set versus older data, you can quantify the drop in predictive accuracy and confirm that the model fails to generalize to the new data distribution. This approach is a standard method for detecting concept drift in machine learning models, especially when external changes (like a loyalty program) alter the underlying patterns.
Exam trap
Salesforce often tests the misconception that improving model performance (e.g., by adding more data or simplifying features) is the correct response to poor generalization, rather than first validating the hypothesis of concept drift through a time-based holdout evaluation.
How to eliminate wrong answers
Option B is wrong because excluding new customers entirely would remove the very data needed to detect the generalization failure, and it does not validate the hypothesis about model performance on new customers. Option C is wrong because increasing training data with older transactions would only reinforce the model's bias toward pre-loyalty-program patterns, making it even less adaptable to the new behavior. Option D is wrong because removing the 'Product Category' feature simplifies the model but does not address the root cause of concept drift; it may reduce accuracy further and does not test whether the loyalty program caused the shift.