A test watched daily and stopped the moment it reaches significance will reach significance eventually whether or not there is an effect. Every look is another chance for random variation to cross the line, and stopping at the first crossing keeps only the crossings.
The defence is deciding the sample size in advance and looking once. That number comes from four inputs, and none of them is the result.
What determines the sample size?
Four things, agreed before the test starts.
| Input | Typical value |
|---|---|
| Baseline rate | whatever you measure today |
| Smallest effect worth detecting | a business decision |
| Significance level | 5% |
| Power | 80% |
Power is the one people skip and it is the important one. It is the probability of detecting an effect that genuinely exists, so a test at 80 per cent power misses one real effect in five — which is a high failure rate for something people treat as definitive.
The smallest effect worth detecting is a decision rather than a measurement. It is the improvement below which you would not act, and setting it honestly is what keeps the sample size realistic.
Why do small effects cost so much?
Because the required sample grows with the inverse square of the effect size. Halving the effect you want to detect roughly quadruples the number of observations needed.
That is the arithmetic behind most abandoned tests. Detecting a twenty per cent improvement might take days; detecting a two per cent one on the same traffic can take longer than the feature will exist.
It is also the argument for testing bigger changes. A test that cannot reach its sample size in a reasonable time was not a viable test, and running it anyway produces an inconclusive result dressed up as a negative one.
What does a confidence interval add?
Precision, which a yes-or-no answer hides. An interval says the effect is somewhere in a range, and the width of that range tells you how much the test actually established.
A result reported as "significant, plus three per cent" and one reported as "somewhere between plus one and plus twelve" are the same finding described honestly and dishonestly. The second makes clear that the true effect could be small.
It also handles the case people find hardest: a non-significant result with a narrow interval around zero is real evidence of no meaningful effect, while a non-significant result with a wide interval is no evidence of anything.
What is the peeking problem exactly?
That the stated error rate assumes one analysis. Testing at five per cent means a one-in-twenty chance of a false positive per look — so twenty looks make a false positive close to expected rather than unlikely.
The honest fixes are to look once at the planned end, or to use a method designed for repeated analysis, which adjusts the threshold to account for the looks. Both are fine; alternating between them after seeing the data is not.
The related trap is stopping a losing test early. That is the same problem in reverse and it is more defensible, because the cost of continuing is real — but it should be a documented rule rather than a reaction.
What else invalidates a result?
Splitting the data after the fact. Finding an effect in one segment of a test that showed nothing overall is nearly always noise, because a test large enough to detect an effect overall is far too small to detect one within a tenth of itself.
Changing anything mid-test does the same. A new traffic source, a seasonal event or a bug fix during the run means the two groups were not comparable throughout.
Running many tests at once has the same shape as peeking. Twenty simultaneous tests at five per cent will produce about one false positive, which is why a surprising win from a minor change deserves a repeat before it is believed.
How long should a test run?
At least a full week, and a whole number of weeks, regardless of what the sample size says. Behaviour differs by day, so a test that starts on a Tuesday and ends on a Friday has weighted weekdays against weekends.
Reaching the required sample in three days is not a reason to stop at three days. It is a reason to keep going to the end of the week and then stop, because the extra data costs nothing and removes a systematic bias.
The upper bound is practical rather than statistical. A test running for months accumulates changes around it — seasonality, other releases, a different traffic mix — until the two groups are no longer comparable.
Questions people ask
Is 80 per cent power enough? It is the convention. For an expensive decision, higher is worth the extra sample.
Can I reuse a sample size from last time? Only if the baseline rate has not moved. It usually has.
What if I cannot reach the number? Test a bigger change, or accept that the question cannot be answered with this traffic.
Does significance mean it matters? No. It means the effect is unlikely to be zero, which the p-value article covers in full.
Work out the number first. The sample size calculator turns a baseline, a detectable effect and a power level into the observations you need, and the confidence interval calculator reports the range once the data is in.