DRV-022

Why your A/B test needs so many visitors

The inverse-square relationship, which is the actual answer to the question behind almost every sample-size search.

The setup

Comparing two proportions, you need enough data that a real difference of size δ is unlikely to be missed and a difference of zero is unlikely to be imagined. Those two requirements are the significance level and the power, and the sample size follows from both.

The derivation

Step 1: Combine the two error rates

The z-quantile for the significance level and the z-quantile for the power add, then square.

(z_alpha/2 + z_beta)^2

Step 2: Scale by the variance of the outcome

A proportion near 0.5 is the noisiest and needs the largest sample; a proportion near 0 or 1 is quieter.

p_bar * (1 - p_bar)

Step 3: Divide by the squared effect

Two arms, so a factor of two, and the effect enters squared. This is the term that dominates everything.

n = 2 * (z_alpha/2 + z_beta)^2 * p_bar * (1 - p_bar) / delta^2

Checks that prove it is right

A formula you cannot test is a formula you have to trust. These take seconds, and they are what separate a derivation from a formula restatement.

Halve δ → Quadruple n

The effect enters squared. This single sentence answers the question behind most sample-size searches, which is not "what is n" but "why is n so large".

δ → 0 → n → ∞

Detecting an arbitrarily small difference requires an arbitrarily large sample. A calculator that returns a finite number here is wrong.

p → 0 or 1 → n falls

A near-certain outcome has little variance to see through. This is why conversion tests on very low base rates need less data than intuition suggests, though the absolute number of conversions is still tiny.

Worked example

A 5% baseline, detecting a 1 percentage point improvement, at 95% confidence and 80% power.

z at 95% two-tailed is 1.96 and z at 80% power is 0.8416, summing to 2.8016 whose square is 7.849. With p̄ = 0.05 the variance term is 0.0475. Dividing by 0.01² gives 7,457 per arm, 14,914 in total. Ask instead for a 0.5 percentage point effect and the requirement becomes 29,826 per arm. Four times as many, from halving the effect.

Quantiles from an independent implementation of the inverse normal CDF, cross-checked against Python NormalDist.

What it assumes

Two arms of equal size.
A normal approximation to the binomial, which holds when np and n(1−p) both exceed about 10.
One test, analysed once. Peeking repeatedly invalidates the significance level entirely.

Where the formula stops being valid

Stating limits plainly is more useful than pretending there are none, and it is the item competitors most consistently omit.

Sequential testing. If you look at the results as they accumulate and stop when significant, the true false-positive rate is far above the nominal one and this formula does not apply.
Very low base rates, where the normal approximation fails and an exact binomial or Poisson treatment is needed.
Multiple variants. Testing four arms against a control needs a correction for the multiple comparisons, which this does not include.