Does a p value below 0.05 mean the variant works?

No. It means data this extreme would be unlikely if the variant did nothing. The probability that the variant is actually better is a different quantity and needs a different method.

Can I stop the test as soon as it goes significant?

No, and doing so is the most common way A/B tests produce false wins. Checking repeatedly and stopping at the first significant reading inflates the false positive rate well beyond the nominal five percent.

Businessmarketing

A/B Test Significance Calculator

A/B test significance by a two proportion z test. The p value is the probability of seeing data this extreme if the variant made no difference, which is not the probability that the variant works, and reading it the second way is how underpowered tests ship as wins.

Also called: ab test calculator, split test significance.

P value

2.83%

The variant converts at 6% against 5%, a relative lift of 20%. The p value is 0.02829, so the result is significant at 95% confidence. The p value is the probability of a gap this large if the variant changed nothing. It is not the probability that the variant works. Detecting a lift this size at 80% power needs about 8156 per arm, and you have 5000, so the test is underpowered.

Control conversion rate

Variant conversion rate

Absolute lift

Relative lift

20%

Z score

2.19

Confidence interval on the difference

0.107% to 1.893%

Sample needed per arm to detect this lift

8,156

P value

0.02829

Verdict

significant

What the p value means

The p value is the probability of a gap this large if the variant changed nothing. It is not the probability that the variant works.

On power

Detecting a lift this size at 80% power needs about 8156 per arm, and you have 5000, so the test is underpowered.

How this is calculated

Method and background

How this is calculated

The test compares two conversion rates against the variation you would expect from sampling alone. A small p value says the observed gap would be unlikely if nothing were happening. It does not say how likely the variant is to be better, which is a different question requiring different machinery. Two other things decide whether a result means anything: whether the test was powered to detect a lift this size in the first place, and whether it ran for whole weeks, since traffic composition changes by day.

a two proportion z test, where the pooled rate estimates the standard error under the null

p: Conversion rates
n: Sample sizes

Worked examples

Each of these is asserted on every build. If a change to the engine ever moved one of these answers, the build would fail before the page could print it.

a one point lift on five thousand each

Control visitors: 5,000
Control conversions: 250
Variant visitors: 5,000
Variant conversions: 300
Confidence level: 95%
Two tailed test: Yes

P value2.83%

250/5000 against 300/5000, a fifth better in relative terms

Open this example

identical arms give no evidence

Control visitors: 5,000
Control conversions: 250
Variant visitors: 5,000
Variant conversions: 250
Confidence level: 95%
Two tailed test: Yes

P value100%

boundary: no difference at all gives a p value of one

Open this example

Method and limits

What it assumes

Independent visitors randomly assigned, each counted once.

What it deliberately does not model

Stopping the moment significance appears inflates the false positive rate substantially, which is the most common way tests mislead.
A statistically significant lift can be too small to matter commercially.
Multiple simultaneous tests need a correction that this does not apply.

Formula version 1.0.0 · definition 1.0.0 · United States · Report a problem with this calculator

Frequently asked questions

Does a p value below 0.05 mean the variant works?: No. It means data this extreme would be unlikely if the variant did nothing. The probability that the variant is actually better is a different quantity and needs a different method.
Can I stop the test as soon as it goes significant?: No, and doing so is the most common way A/B tests produce false wins. Checking repeatedly and stopping at the first significant reading inflates the false positive rate well beyond the nominal five percent.