Without hypothesis testing, you would have no way to tell whether a difference in your data is real or just random noise — meaning you could waste money on changes that do nothing, or miss problems that are actually there. For the DA0-002 exam, you need to know how to make these decisions statistically and how to construct confidence intervals that communicate the uncertainty around your estimates. This chapter gives you the framework to answer those exam questions with confidence.
Jump to a section
A simple way to picture Hypothesis Testing and Confidence Intervals
72 grams is the exact amount of coffee your favourite café claims their industrial grinder dispenses per shot. You want to know if their new barista is actually hitting that target, because your morning depends on it. You cannot weigh every single shot they make over the next year — that would be impractical. Instead, you collect a sample: you weigh 30 random shots over a week. The average weight of your sample is 69.8 grams. Is that close enough to 72 grams to be just random luck, or is the new barista systematically under-dosing?
This is hypothesis testing. You start with a null hypothesis: "The true average is 72 grams; any difference is just random noise." Your alternative hypothesis is "The true average is not 72 grams." You set a threshold — a significance level — say 5%. If the probability of seeing a 69.8-gram average by pure chance is less than 5%, you reject the null hypothesis and conclude the barista is off.
But you also want a range. You calculate: "I am 95% confident the true average dose is between 69.2 and 70.4 grams." That range is your confidence interval. It quantifies the uncertainty. The narrower the interval, the more precise your estimate. The café owner uses this to decide whether to re-train the barista or recalibrate the grinder. Every step maps exactly to what a data analyst does with business data.
Hypothesis testing and confidence intervals are two sides of the same coin. They give you a disciplined way to make decisions from data without guessing or jumping to conclusions. Let us start from the very beginning.
A hypothesis is simply an assumption you are testing. In statistics, we frame it as two competing statements. The null hypothesis (H0) is the default position — usually that there is no change, no effect, or no difference. For example, "The average response time of the server is 200 milliseconds." The alternative hypothesis (H1 or Ha) is what you suspect might be true — "The average response time is not 200 milliseconds" (two-tailed), or "The average response time is greater than 200 milliseconds" (one-tailed).
The next key concept is the significance level, denoted by the Greek letter alpha (α). This is the risk you are willing to take of being wrong when you reject the null hypothesis. A common alpha is 0.05, meaning you accept a 5% chance of concluding there is an effect when there actually is none. This kind of error is called a Type I error — a false positive. The other kind of error, Type II error (beta), is failing to detect an effect that is really there (a false negative).
To perform a hypothesis test, you collect a sample — a subset of the entire population (the complete set you care about). From that sample, you calculate a test statistic, which is a standardised value that measures how far your sample result is from the null hypothesis, taking into account the variability in your data. The test statistic could be a t-statistic, z-statistic, chi-square statistic, or F-statistic, depending on your data type and question.
The test statistic is then compared to a critical value from a theoretical distribution, or more commonly, you compute a p-value. The p-value is the probability of observing your sample data, or something more extreme, if the null hypothesis were true. If the p-value is less than your chosen alpha (say 0.05), you reject the null hypothesis. If it is greater, you fail to reject the null — you do not have enough evidence.
Now, confidence intervals approach the same problem from a different angle. Instead of a binary decision (reject or not), a confidence interval gives you a range of plausible values for the population parameter (like the mean). A 95% confidence interval means that if you repeated the sampling process many times, about 95% of those intervals would contain the true population parameter. It does not mean there is a 95% probability that the true value lies in your particular interval — that is a common misunderstanding.
To build a confidence interval for a mean, you need three things:
The sample mean (the average of your data)
The standard error (how spread out the sample means would be if you took many samples — it is the sample standard deviation divided by the square root of the sample size)
The critical value from a t-distribution or z-distribution (depending on sample size and whether you know the population standard deviation)
The formula is: sample mean ± (critical value × standard error).
Why does this replace gut-feel decisions? Before modern inferential statistics, people relied on rules of thumb. For example, "If the new sales strategy yields 10% more revenue, we will adopt it." But that ignores variation: some months are naturally better, and random chance can produce a 10% difference. Hypothesis testing and confidence intervals formalise this intuition, accounting for sample size and variability.
For the DA0-002 exam, you do not need to perform complex calculations by hand. You need to interpret output from statistical software and understand the logic. The exam will present you with scenarios and ask: "Based on the p-value shown, should the analyst reject the null hypothesis?" Or: "What does a 95% confidence interval of (12.3, 15.7) tell us?" The answer always hinges on comparing the p-value to alpha, or checking whether a claimed value falls inside the interval.
One more important concept: the one-tailed versus two-tailed test. A one-tailed test checks for an effect in one direction only (greater than or less than). A two-tailed test checks for any difference. Two-tailed tests are more conservative and are used when you have no strong prior expectation about the direction of the effect. The exam likes to test whether you know which to choose based on the research question.
Finally, power is the probability that your test will detect an effect if one truly exists. Power depends on sample size, effect size (how big the difference is), alpha, and variability. Larger samples and larger effects give more power. The exam may ask you to identify ways to increase power, for example by increasing the sample size.
Define the null and alternative hypotheses
State H0 as the default assumption (e.g., mean = 100) and Ha as what you want to prove (e.g., mean ≠ 100 for two-tailed, or mean > 100 for one-tailed). This sets the framework for the entire test.
Choose the significance level (alpha)
Pick a threshold for risk, commonly 0.05. This is the probability you are willing to accept of making a Type I error. The exam will almost always give you alpha explicitly.
Collect a sample and calculate the test statistic
Gather data and compute a relevant test statistic (t, z, chi-square, or F) depending on your data type. The test statistic measures how far your sample result is from H0 in terms of standard error.
Compute the p-value
The p-value comes from the test statistic and its distribution. It tells you the probability of observing your data (or more extreme) if H0 is true. A small p-value means the data are unlikely under H0.
Compare p-value to alpha and make a decision
If p < alpha, reject H0 — there is evidence for Ha. If p ≥ alpha, fail to reject H0 — not enough evidence. Never say 'accept H0'.
Construct the confidence interval
Calculate the sample statistic plus/minus the margin of error (critical value × standard error). This gives a range of plausible values for the population parameter at a chosen confidence level (e.g., 95%).
Interpret results in context
State the conclusion in plain language. For example: 'We have statistical evidence that the new process reduces errors (p = 0.03), and we are 95% confident the true reduction is between 2% and 5%.'
Imagine you are a junior data analyst at an online retail company. The marketing team has launched a new website layout hoping to increase the average order value. The old layout had an average order value of £45. After one month with the new layout, you have data from 500 transactions with an average of £47.50. Is the new layout better, or is this just random fluctuation?
You would set up the following hypothesis test:
Null hypothesis: The true average order value with the new layout is £45.
Alternative hypothesis: The true average order value with the new layout is greater than £45 (one-tailed, because you only care about an increase).
Significance level: α = 0.05.
You run a one-sample t-test using software like Excel, Python, R, or a dedicated statistical tool. The output shows a test statistic of 2.10 and a p-value of 0.018. Since 0.018 < 0.05, you reject the null hypothesis. You conclude there is statistical evidence that the new layout increases average order value.
Next, you construct a 95% confidence interval for the true mean order value under the new layout. The software tells you the interval is (£46.20, £48.80). This means you are 95% confident that the population mean order value for the new layout lies somewhere between £46.20 and £48.80. Crucially, the entire interval is above £45, which aligns with your hypothesis test result.
What happens next? You present these results to the marketing team. But you also point out that while the increase is statistically significant, it might not be practically significant — an extra £2.50 per order may or may not justify the effort and cost of redesigning the site. The confidence interval gives them the range to make that business judgement.
In practice, IT professionals use hypothesis testing for:
A/B testing: comparing two versions of a webpage, email subject line, or app feature.
Quality assurance: checking whether a batch of manufactured components meets specifications.
Performance monitoring: testing if a server upgrade has reduced response times.
User behaviour analysis: seeing if a new feature changes user engagement metrics.
When presenting findings, you would include both the p-value and the confidence interval. The p-value tells you whether the result is statistically significant. The confidence interval tells you the magnitude of the effect and its precision. Together, they give a complete picture.
The DA0-002 exam tests hypothesis testing and confidence intervals in several specific ways. You do not need to calculate critical values by hand, but you must understand the concepts thoroughly.
Key exam topics:
Interpreting p-values: You will see a p-value and need to compare it to a given alpha (usually 0.05). If p < alpha, reject H0. If p > alpha, fail to reject H0. The exam loves to ask this explicitly.
Confidence interval interpretation: A 95% confidence interval of (10, 20) means you are 95% confident the population parameter lies between 10 and 20. Watch out for the trap that says "95% of the data lies in this range" — that is wrong.
The relationship between hypothesis tests and confidence intervals: If a claimed value falls outside the 95% confidence interval, then a two-tailed hypothesis test at alpha = 0.05 would reject that value. The exam may ask you to make this connection.
One-tailed vs two-tailed tests: You must choose correctly based on the question. If the question says "test whether the new system is faster", that is a one-tailed test (greater than). If it says "test whether the new system is different", that is two-tailed.
Type I and Type II errors: The exam will ask what each error is and how to reduce them. Remember: Type I = false positive (rejecting a true H0). Type II = false negative (failing to reject a false H0). Lowering alpha reduces Type I but increases Type II. Increasing sample size reduces both.
Statistical significance vs practical significance: A result can be statistically significant (low p-value) but practically meaningless (the effect is tiny). The exam tests whether you recognise this distinction.
Common trap patterns:
Trap: Interpreting a confidence interval as containing the population mean with 95% probability. Remember: the interval captures the parameter 95% of the time across repeated samples, not a probability for a single interval.
Trap: Saying "fail to reject the null" means the null is true. It does not. It only means you do not have enough evidence against it.
Trap: Forgetting that the significance level is chosen before seeing the data. The exam might present a scenario where an analyst adjusts alpha after looking at results — that is bad practice.
Key definitions to memorise:
Null hypothesis (H0): the default assumption of no effect.
Alternative hypothesis (H1/Ha): what you want to prove.
p-value: probability of observing data as extreme as yours, assuming H0 is true.
Confidence level: the percentage of confidence intervals that would contain the true parameter (e.g., 95%).
Standard error: variability of the sample statistic (e.g., sample mean).
Effect size: the magnitude of the difference, independent of sample size.
Make sure you can distinguish between a parameter (a characteristic of the whole population, like the true mean) and a statistic (a characteristic of your sample, like the sample mean). The exam tests this distinction frequently.
Finally, know that the output from a t-test includes a p-value and confidence interval by default. You will need to extract the right numbers and interpret them in a business context.
The p-value is the probability of seeing your data if the null hypothesis is true, not the probability that the null hypothesis is true.
If the p-value is less than your significance level (alpha), you reject the null hypothesis; otherwise, you fail to reject it.
A 95% confidence interval means that if you repeated the sampling process many times, 95% of those intervals would contain the true population parameter.
Type I error (false positive) is rejecting a true null hypothesis; Type II error (false negative) is failing to reject a false null hypothesis.
Increasing sample size reduces both Type I and Type II errors and narrows confidence intervals, giving more precise estimates.
Statistical significance does not equal practical significance — a tiny effect can be significant with a large enough sample.
The confidence interval and the two-tailed hypothesis test at the same confidence level will always agree: if the null value lies outside the interval, reject H0.
One-tailed tests have more power to detect an effect in one direction but cannot detect an effect in the opposite direction; choose based on the research question.
The standard error measures the variability of a sample statistic; it decreases as sample size increases.
You cannot 'accept' the null hypothesis — you can only fail to reject it based on the available evidence.
These come up on the exam all the time. Here's how to tell them apart.
Null Hypothesis (H0)
Default assumption of no effect or no difference
What you are trying to disprove
Often stated as something equals a specific value
Alternative Hypothesis (Ha)
What you suspect might be true
What you are trying to find evidence for
Often stated as something is not equal, greater than, or less than a value
Type I Error
False positive: rejecting a true null hypothesis
Probability is denoted by alpha (α)
Can be reduced by lowering the significance level
Type II Error
False negative: failing to reject a false null hypothesis
Probability is denoted by beta (β)
Can be reduced by increasing sample size or effect size
Statistical Significance
Determined by p-value compared to alpha
Tells you if an effect exists (not how big it is)
Can be achieved with a large sample even for tiny effects
Practical Significance
Judged by the real-world impact of the effect size
Considers cost, time, and business value
A statistically significant result may not be worth implementing
Parameter
Describes the entire population (e.g., true population mean)
Usually unknown and estimated from a sample
Symbolised by Greek letters (e.g., μ, σ)
Statistic
Describes a sample (e.g., sample mean)
Used to estimate the parameter
Symbolised by Roman letters (e.g., x̄, s)
One-Tailed Test
Tests for effect in one direction only (greater than or less than)
More power to detect an effect in that direction
Critical region is in one tail of the distribution
Two-Tailed Test
Tests for any difference, regardless of direction
More conservative, requires larger difference to be significant
Critical region is split between both tails
p-value
A single number (probability) between 0 and 1
Used for binary decision: reject or fail to reject H0
Does not convey the range of plausible effect sizes
Confidence Interval
A range of values (e.g., 10 to 20)
Provides an estimate of the effect size and its precision
Gives a sense of uncertainty around the estimate
Mistake
A p-value tells you the probability that the null hypothesis is true.
Correct
The p-value assumes the null hypothesis is true and tells you the probability of seeing your data (or something more extreme) under that assumption. It does not give the probability that H0 is correct.
It feels intuitive to want a direct probability statement about H0, but the p-value works the other way around.
Mistake
If the p-value is greater than 0.05, the null hypothesis is proven true.
Correct
A high p-value means you do not have enough evidence to reject H0. It does not prove H0 is true; the test may lack power or the true effect may be very small.
People want a binary answer — accept or reject — but statistics only gives evidence for rejection, not for acceptance.
Mistake
A 95% confidence interval contains 95% of the data points.
Correct
A 95% confidence interval is about the population mean (or another parameter), not about individual data points. It gives a range where the true mean likely lies, not where most observations fall.
The word 'confidence' sounds like 'probability' about individual values, but the target is the parameter, not the data points.
Mistake
Hypothesis testing can prove that a new treatment works.
Correct
Hypothesis testing can only provide evidence against the null. It never proves the alternative is true; it only supports it with a certain level of confidence. A single study is never definitive proof.
The word 'test' implies a definitive verdict, but in statistics, conclusions are always probabilistic and subject to uncertainty.
Mistake
The significance level alpha is the probability of making a Type II error.
Correct
Alpha is the probability of making a Type I error (rejecting a true H0). The probability of a Type II error is called beta (β).
Both errors are easy to confuse because they sound similar. Many beginners mix up which Greek letter corresponds to which error.
Mistake
If a confidence interval is wide, the data must be very variable or the sample size is small. But a wide interval is always bad.
Correct
A wide interval does indicate high variability or low sample size, which is less precise. However, sometimes a wide interval is honest about uncertainty — a narrow interval from a biased sample can be misleading.
People want precision, so they instinctively prefer narrow intervals, but accuracy matters more than precision. A wide confidence interval might be the correct reflection of limited data.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
A one-tailed test checks for an effect in one direction (greater than or less than). A two-tailed test checks for any difference regardless of direction. Use one-tailed if you only care about an increase (or only a decrease); use two-tailed if you want to detect any change.
Yes, but the interval will be wider because you need more certainty. A 99% confidence interval includes more possible values than a 95% interval, reflecting the higher confidence level.
If the confidence interval for a difference (e.g., mean1 minus mean2) includes zero, you cannot be confident that there is a real difference — the effect could be zero. A hypothesis test would fail to reject the null at the corresponding alpha.
Use a t-test when your sample size is small (typically under 30) or when you do not know the population standard deviation. Use a z-test for large samples or when the population standard deviation is known.
The margin of error is the 'plus or minus' part of the interval. It equals the critical value multiplied by the standard error. A larger margin of error means a wider interval and less precise estimate.
Alpha is a business or scientific decision. 0.05 is standard for most research. Use a smaller alpha (e.g., 0.01) when the cost of a false positive is high, like in medical trials. Use a larger alpha (e.g., 0.10) when exploratory work allows more risk.
You've finished Hypothesis Testing and Confidence Intervals. Continue through the DA0-002 study guide to build a complete picture of the exam.
Done with this chapter?