A/B test significance and sample size calculator
Two modes. Results tells you whether a winner is real or noise. Planner tells you how much traffic a test needs before it can tell you anything at all.
How it is calculated
Results uses a two-sided, two-proportion z-test, the standard test for comparing conversion rates.
Conversion rate p = conversions ÷ visitors Pooled rate p̄ = (conv_A + conv_B) ÷ (n_A + n_B) z = (p_B − p_A) ÷ √( p̄(1 − p̄)(1/n_A + 1/n_B) ) p-value = 2 × (1 − Φ(|z|)) Sample per variant n = ( z₁₋α/₂·√(2p̄(1−p̄)) + z₁₋β·√(p₁(1−p₁) + p₂(1−p₂)) )² ÷ (p₂ − p₁)²
How to use the result
Decide the sample size before the test starts and do not stop early because a result looks significant on day three; peeking inflates false positives. For paid media creative tests, use the same discipline with a holdout, and judge winners on cost per purchase, not click-through rate.
Questions
What does statistically significant mean in an A/B test?
It means the difference between variants is unlikely to be random noise at your chosen confidence level. At 95% confidence, a result this large would appear by chance less than 5% of the time if the variants were actually identical.
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest uplift you care about detecting. At a 2% baseline, detecting a 10% relative uplift with 80% power at 95% confidence needs roughly 80,000 visitors per variant.
What is minimum detectable effect?
Minimum detectable effect (MDE) is the smallest relative improvement the test is designed to detect reliably. Smaller MDEs need much more traffic: halving the MDE roughly quadruples the sample size.
Can I stop a test as soon as it is significant?
No. Checking repeatedly and stopping at the first significant reading greatly increases false positives. Fix the sample size in advance, or use a sequential testing method designed for early stopping.
If growth is the conversation, let’s have it.
Advisory conversations, speaking, podcasts and media. Tell me what you are working on and I will reply personally within two working days.