An analyst runs an A/B test with 1000 users per group and observes a conversion rate of 5% in the control and 6% in the treatment. The p-value is 0.12. What should the analyst conclude?
With a p-value of 0.12 exceeding the 0.05 significance threshold, the observed 1% conversion lift is within the range expected from random variation. The analyst cannot reject the null hypothesis, so the difference lacks statistical significance.
Why this answer
A p-value of 0.12 is greater than the conventional significance level of 0.05, so the null hypothesis of no difference cannot be rejected. The observed difference between 5% and 6% conversion rates is not statistically significant at the 0.05 level. The analyst should conclude that there is insufficient evidence to claim the treatment outperforms control.
Exam trap
DA0-002 often tests the misinterpretation of p-values, especially the false belief that a non-significant result proves no effect or that p-value equals the probability the treatment is better.
How to eliminate wrong answers
Option B is wrong because the sample size may or may not be too small; the p-value alone does not determine that, and with 1000 per group the test has reasonable power to detect large effects. Option C is wrong because a p-value of 0.12 does not indicate statistical significance; claiming the treatment significantly outperforms control is incorrect. Option D is wrong because a p-value is not the probability that the treatment is better; it is the probability of observing such data if the null hypothesis were true.