Read the definition carefully and note what is on which side of the "if". You are being told how surprising the data would be in a world where the null hypothesis holds — not how likely that world is, and not the chance that your finding is mistaken. The conditional runs the opposite way from the one almost everybody hears. The familiar 1.96 falls out of the same definition: it is the z-score whose two tails together account for five per cent of a normal distribution, which is why it turns up on every statistics course as the threshold.
The gap between those two readings is the source of most of the misuse, and it is a genuine difference rather than a pedantic one.
How do the pieces fit together?
Four quantities, each derived from the one before it.
| Quantity | What it says |
|---|---|
| Standard deviation | Spread of the data around its mean |
| Standard error | Spread of the mean across repeated samples |
| z-score | How many standard deviations a value sits from the mean |
| p-value | Chance of a z-score this extreme under the null |
Subtract the mean and divide by the standard deviation, and what comes out is a count of standard deviations rather than a measurement. A 115 where the mean is 100 and the spread is 15 is exactly one of them out, which puts it at the 84th percentile — a fact about the normal distribution, not about the test. Standardising is what makes different scales comparable: an IQ of 115 and a test score one standard deviation above its own mean are the same claim expressed in different units.
Which standard deviation should you use?
The sample version; dividing by n−1; unless you have measured the entire population, which almost nobody has. The population version divides by n and understates the spread of the group the sample came from.
The correction matters most on small samples: with eight values it is about 7 per cent, with eighty around 0.6 per cent, and with eight hundred it is negligible. The choice should follow what the numbers are, not which figure looks better.
The averages article covers the related decision about which centre to report alongside it.
What does a confidence interval actually claim?
Less than it sounds. A 95 per cent confidence interval does not mean there is a 95 per cent chance the true value lies inside this particular interval: it means that if you repeated the sampling procedure many times, 95 per cent of the intervals produced would contain the true value.
A mean of 100 with a standard deviation of 15 and n = 100 gives a 95 per cent interval of roughly 97.06 to 102.94. The interval is a property of the method, and the true value is either inside this one or it is not.
The practical value of that distinction is that it stops the interval being read as a probability distribution over the answer. It is a statement about long-run behaviour of the procedure, which is a weaker and more honest claim.
How many people do you need to survey?
For a proportion, z² × p(1−p) ÷ margin². At 95 per cent confidence with a ±5 per cent margin and no prior expectation, that is 385 responses, and it barely changes with population size unless the population is small.
That last point surprises people. Sampling 385 from a city of a million and from a town of fifty thousand gives essentially the same margin, because the margin depends on the sample size rather than the fraction of the population sampled.
What the calculation does not cover is far larger than what it does. It answers sampling error only: the randomness of who happened to respond, and says nothing about non-response bias, question wording, or a sampling frame that never included the people whose answers would have changed the result.
Where does a p-value get abused?
Three well-documented ways, all of which produce smaller numbers without any additional evidence.
- Choosing one-tailed after seeing the direction. Halves the p-value for free, and is only legitimate if the direction was specified in advance.
- Testing many hypotheses and reporting the significant one. At p < 0.05, one test in twenty passes by chance alone.
- Stopping when it crosses the threshold. Optional stopping inflates the false positive rate substantially.
None of these requires bad faith and all of them are common. The defence is pre-registration of the analysis, or at minimum stating how many tests were run.
Questions people ask
Is p < 0.05 a law? It is a convention, chosen by Fisher as a round figure and never intended as a bright line. A p of 0.049 and one of 0.051 are essentially the same evidence.
Does a small p mean a large effect? No. With a large enough sample, a trivially small effect produces a tiny p-value. Effect size and significance are separate questions and both should be reported.
What is the difference between standard deviation and standard error? Standard deviation describes the data; standard error describes the precision of an estimate made from it, and it shrinks as the sample grows.
Why 1.96? Because 95 per cent of a normal distribution lies within 1.96 standard deviations of the mean. It is a property of the distribution rather than a chosen number.
Should I report a confidence interval or a p-value? The interval, where you can. It carries the effect size and the precision together, where a p-value collapses both into one number that answers neither question directly.
The p-value answers a narrow question precisely, and a broad one not at all. The p-value calculator and z-score calculator handle the standardising step, the standard deviation calculator gives the spread underneath it, and the confidence interval and sample size calculators cover what a survey can and cannot claim.