In an A/B test, significance at 95% means a difference this large would appear less than 5% of the time if the variants were truly identical. It does not tell you the size of the effect is reliable to the decimal point, and checking repeatedly and stopping at the first significant reading greatly increases false positives.
Formula
z = (p_B − p_A) ÷ √( p̄(1 − p̄)(1/n_A + 1/n_B) )
Check whether a test result is real, or plan how many visitors and days a test needs before you start.
Related terms
- Minimum detectable effect (MDE)Minimum detectable effect is the smallest relative improvement a test is designed to detect reliably.
- Holdout test (geo lift)A holdout test withholds advertising from a randomly chosen group of people or regions and compares their results with a group that saw the ads.
- Conversion rateConversion rate is the share of visitors or clicks that complete a target action, such as a purchase.
Part of the growth marketing glossary by Shiva Sharma.