ADSSystem

A/B test sample size calculator

How many visitors each variant needs before the result means anything.
In one line

Works out how many visitors per variant an A/B test needs to detect a given lift, and how many days that takes at your traffic.

n per variant = (z confidence + z power)squared x [p1(1-p1) + p2(1-p2)] / (p2 - p1)squared
Where the control sits today.
Relative, not absolute. 15 means 3.20% moving to 3.68%.
90, 95 or 99. Power is fixed at 80%.
Visitors per variant-
Total visitors needed-
Days to run-
Detecting-
Visitors per variant at a 3% baseline95% confidence, 80% power207,9005% lift53,20010% lift24,20015% lift13,90020% lift6,45030% lift30 DAYS AT 1,200 A DAY

The formula

n per variant = (z confidence + z power)squared x [p1(1-p1) + p2(1-p2)] / (p2 – p1)squared

p1 is the baseline conversion rate, p2 is the rate you would get if the change worked. The z values come from the two error rates you are willing to accept: 1.96 for 95 percent confidence, 0.84 for 80 percent power.

The shortcut worth remembering is that sample size scales with the inverse square of the effect. Halve the lift you want to detect and the test gets roughly four times longer. That single fact kills most test plans before they start.

The lift is decided before the test starts

Most teams pick a variant, launch it, and find out afterwards what it could have proved. That is backwards. The minimum detectable effect is an input, not a result.

  • Decide the smallest lift that would change what you do next.
  • If a 5 percent lift would not change the decision, do not pay for the sample size to detect it.
  • If the sample size for that lift exceeds a month of traffic, the test is not available to you. Test something bigger.

Low traffic sites are not barred from testing. They are barred from testing small changes. Button colours need millions of visitors. Rewriting an offer does not.

Confidence and power are two different mistakes

Confidence controls false positives: shipping a change that did nothing. Power controls false negatives: binning a change that worked. Almost every calculator lets you set the first and quietly fixes the second at 80 percent.

Eighty percent power means one in five real improvements will be missed. That is the accepted trade, but it is a trade, and it is worth knowing you made it.

The trap

The sample size is a commitment, not a suggestion.

The number this page gives you only holds if you look once, at the end. Checking daily and stopping the moment the result goes green turns a 5 percent false positive rate into something closer to 20 or 30 percent, depending on how often you peek.

This is why so many winning tests fail to replicate in revenue. The test did not lie. The stopping rule did.

Frequently asked

It depends on the baseline conversion rate and the lift you want to detect. At a 3 percent baseline and a 15 percent relative lift, roughly 24,000 visitors per variant at 95 percent confidence and 80 percent power.
The smallest lift that would change a decision. Anything smaller is expensive to measure and irrelevant once measured.
No. Repeated checking with an early stop inflates the false positive rate well above the 5 percent the test assumes. Fix the sample size first, then run it to the end.
Not directly. This formula assumes a binary outcome, converted or not. Continuous metrics like revenue per visitor need the variance of the metric, which is usually much larger and needs a bigger sample.