A p-value is the probability of observing a result at least this extreme if the null hypothesis were true. It is not the probability that the null hypothesis is true, and it is not the probability that your finding is wrong. A z-score of 1.96 gives a two-tailed p of 0.05, which is why 1.96 is the critical value that appears in every statistics course.
The gap between those two readings is the source of most of the misuse, and it is a genuine difference rather than a pedantic one.
How do the pieces fit together?
Four quantities, each derived from the one before it.
| Quantity | What it says |
|---|---|
| Standard deviation | Spread of the data around its mean |
| Standard error | Spread of the mean across repeated samples |
| z-score | How many standard deviations a value sits from the mean |
| p-value | Chance of a z-score this extreme under the null |
A z-score is (x − μ) ÷ σ, so a score of 115 against a mean of 100 and a standard deviation of 15 gives z = 1 — the 84th percentile. Standardising is what makes different scales comparable: an IQ of 115 and a test score one standard deviation above its own mean are the same claim expressed in different units.
Which standard deviation should you use?
The sample version — dividing by n−1 — unless you have measured the entire population, which almost nobody has. The population version divides by n and understates the spread of the group the sample came from.
The correction matters most on small samples: with eight values it is about 7 per cent, with eighty around 0.6 per cent, and with eight hundred it is negligible. The choice should follow what the numbers are, not which figure looks better.
The averages article covers the related decision about which centre to report alongside it.
What does a confidence interval actually claim?
Less than it sounds. A 95 per cent confidence interval does not mean there is a 95 per cent chance the true value lies inside this particular interval — it means that if you repeated the sampling procedure many times, 95 per cent of the intervals produced would contain the true value.
A mean of 100 with a standard deviation of 15 and n = 100 gives a 95 per cent interval of roughly 97.06 to 102.94. The interval is a property of the method, and the true value is either inside this one or it is not.
The practical value of that distinction is that it stops the interval being read as a probability distribution over the answer. It is a statement about long-run behaviour of the procedure, which is a weaker and more honest claim.
How many people do you need to survey?
For a proportion, z² × p(1−p) ÷ margin². At 95 per cent confidence with a ±5 per cent margin and no prior expectation, that is 385 responses — and it barely changes with population size unless the population is small.
That last point surprises people. Sampling 385 from a city of a million and from a town of fifty thousand gives essentially the same margin, because the margin depends on the sample size rather than the fraction of the population sampled.
What the calculation does not cover is far larger than what it does. It answers sampling error only — the randomness of who happened to respond — and says nothing about non-response bias, question wording, or a sampling frame that never included the people whose answers would have changed the result.
Where does a p-value get abused?
Three well-documented ways, all of which produce smaller numbers without any additional evidence.
- Choosing one-tailed after seeing the direction. Halves the p-value for free, and is only legitimate if the direction was specified in advance.
- Testing many hypotheses and reporting the significant one. At p < 0.05, one test in twenty passes by chance alone.
- Stopping when it crosses the threshold. Optional stopping inflates the false positive rate substantially.
None of these requires bad faith and all of them are common. The defence is pre-registration of the analysis, or at minimum stating how many tests were run.
Questions people ask
Is p < 0.05 a law? It is a convention, chosen by Fisher as a round figure and never intended as a bright line. A p of 0.049 and one of 0.051 are essentially the same evidence.
Does a small p mean a large effect? No. With a large enough sample, a trivially small effect produces a tiny p-value. Effect size and significance are separate questions and both should be reported.
What is the difference between standard deviation and standard error? Standard deviation describes the data; standard error describes the precision of an estimate made from it, and it shrinks as the sample grows.
Why 1.96? Because 95 per cent of a normal distribution lies within 1.96 standard deviations of the mean. It is a property of the distribution rather than a chosen number.
Should I report a confidence interval or a p-value? The interval, where you can. It carries the effect size and the precision together, where a p-value collapses both into one number that answers neither question directly.
The p-value answers a narrow question precisely, and a broad one not at all. The p-value calculator and z-score calculator handle the standardising step, the standard deviation calculator gives the spread underneath it, and the confidence interval and sample size calculators cover what a survey can and cannot claim.