Why do I need so many visitors to detect a small lift?

Because the sample size needed grows with the inverse square of the effect. Halving the lift you want to detect multiplies the sample by four, which is why detecting a 1% relative improvement on a 3% conversion rate needs hundreds of thousands of visitors per arm.

What happens if I stop the test as soon as it looks significant?

You inflate the false positive rate badly, often from 5% to above 20%. Peeking repeatedly at an accumulating result gives many chances to cross the threshold by luck. Fix the sample size in advance, or use a sequential test designed for it.

Businessmarketing

A/B Test Sample Size Calculator

Sample size per arm for a two-proportion test. The inverse-square relationship is the insight: halving the detectable effect quadruples the sample you need.

Also called: how many users for an a b test, split test sample size.

Sample size per arm

7,457

7,457 per arm, 14,914 in total. Halving the effect you want to detect would quadruple this. Sample size scales with the inverse square of the effect, which is the real answer to "why does my test need so many visitors".

Total sample

14,914

z(α/2)

1.96

z(β)

0.8416

How this is calculated

Method and background

How this is calculated

The formula balances two error rates. The chance of calling a difference that is not there, and the chance of missing one that is. At 95% confidence and 80% power the z-terms sum to 2.80, whose square is 7.849. Everything else is the variance of the baseline rate over the square of the effect you want to catch.

n = 2 * (z_alpha + z_beta)^2 * p*(1-p) / delta^2

p: Pooled baseline rate (decimal)
delta: Minimum detectable effect (decimal)

Full derivation: Why your A/B test needs so many visitors

Worked examples

Each of these is asserted on every build. If a change to the engine ever moved one of these answers, the build would fail before the page could print it.

5% baseline, 1pp effect, 95/80

Baseline conversion rate: 5%
Minimum detectable effect (absolute, pp): 1%
Confidence: 95%
Power: 80%

Sample size per arm7,457

DRV-022: standard normal quantiles

Open this example

halving the effect quadruples the sample

Baseline conversion rate: 5%
Minimum detectable effect (absolute, pp): 0.5%
Confidence: 95%
Power: 80%

Sample size per arm29,826

DRV-022 inverse-square relationship; independently computed with Python NormalDist

Open this example

a zero effect is undetectable

Baseline conversion rate: 5%
Minimum detectable effect (absolute, pp): 0%
Confidence: 95%
Power: 80%

Sample size per arm0

degenerate case: an infinite sample, reported as 0 rather than Infinity

Open this example

Frequently asked questions

Why do I need so many visitors to detect a small lift?: Because the sample size needed grows with the inverse square of the effect. Halving the lift you want to detect multiplies the sample by four, which is why detecting a 1% relative improvement on a 3% conversion rate needs hundreds of thousands of visitors per arm.
What happens if I stop the test as soon as it looks significant?: You inflate the false positive rate badly, often from 5% to above 20%. Peeking repeatedly at an accumulating result gives many chances to cross the threshold by luck. Fix the sample size in advance, or use a sequential test designed for it.