DA0-002 Data Analysis Practice Question
An analyst runs an A/B test with 1000 users per group and observes a conversion rate of 5% in the control and 6% in the treatment. The p-value is 0.12. What should the analyst conclude?
⚠ Common exam trap
DA0-002 often tests the misinterpretation of p-values, especially the false belief that a non-significant result proves no effect or that p-value equals the probability the treatment is better.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
The difference is not statistically significant at the 0.05 level.
A p-value of 0.12 is greater than the conventional significance level of 0.05, so the null hypothesis of no difference cannot be rejected. The observed difference between 5% and 6% conversion rates is not statistically significant at the 0.05 level. The analyst should conclude that there is insufficient evidence to claim the treatment outperforms control.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
The difference is not statistically significant at the 0.05 level.
Why this is correct
With a p-value of 0.12 exceeding the 0.05 significance threshold, the observed 1% conversion lift is within the range expected from random variation. The analyst cannot reject the null hypothesis, so the difference lacks statistical significance.
- ✗
The sample size is too small to detect an effect.
Why it's wrong here
With 1000 users per group the test has ample power to detect a 1-point lift; the p-value of 0.12 reflects a genuine absence of evidence, not insufficient sample. It is tempting because small samples often cause non-significance, but here the design is adequately powered.
- ✗
The treatment significantly outperforms control.
Why it's wrong here
A p-value of 0.12 exceeds the conventional 0.05 threshold, so the result is not statistically significant and the null hypothesis is not rejected. It is tempting because the treatment rate is numerically higher, but that observed difference is consistent with chance variation.
- ✗
There is a 12% chance the treatment is better.
Why it's wrong here
A p-value is the probability of observing data this extreme if the null were true, not the probability the treatment is superior. It is tempting because 0.12 looks like a percentage, but it is a conditional probability about the data, not a posterior probability about the hypothesis.
Go deeper
Related to this question
About these practice questions
This DA0-002 question is part of Courseiva's 1,004-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
JA
Written and reviewed by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
Last reviewed September 2026 · checked against the official CompTIA exam blueprint
This DA0-002 practice question is part of Courseiva's free CompTIA certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DA0-002 exam.