ADSSystem

A/B test calculator

Conversion rate for each variant, the lift between them, and whether the difference is real or noise.
In one line

Enter visitors and conversions for both variants to get conversion rates, lift, and whether the result is statistically significant.

z = (pB − pA) ÷ √[ p(1−p) × (1/nA + 1/nB) ]
Confidence -
Variant A rate-
Variant B rate-
Relative lift-
Confidence is the probability the difference is not chance. Below 95% the test has not decided anything.
Two variants, one question5,000 visitors eachA 2.50%B 3.20%CONFIDENCE98.4%The gap between the two curves is wider thanthe noise in either of them. B wins.

The formula

z = (pB − pA) ÷ √[ p(1−p) × (1/nA + 1/nB) ]

How the test decides

The calculator runs a two-tailed z-test on two proportions. It pools the conversion rates of both variants, works out how much random variation you would expect from samples that size, and asks how far the observed gap sits outside that noise. The output is confidence: the probability that a difference this large would not have appeared by chance alone.

  • Ninety-five percent is the conventional threshold, which means you accept a one-in-twenty chance of calling a winner that is not one.
  • That is a business decision, not a law of nature.
  • If the change is cheap to ship and cheap to reverse, 90 percent is defensible.
  • If it rewrites your checkout, 99 percent is not paranoid.

Sample size is the real constraint

Baseline rateTo detect +10% relative
1%~155,000 per variant
2%~77,000 per variant
5%~30,000 per variant
10%~14,000 per variant
20%~6,000 per variant
Rough figures at 95 percent confidence and 80 percent power. The pattern matters more than the precision: low conversion rates and small effects need enormous traffic. Most accounts running a 2 percent conversion rate on 20,000 monthly sessions cannot detect a 10 percent improvement inside a quarter, which means most of their tests were never going to conclude anything.

When the test is inconclusive

An inconclusive test is not a failed test – it is a result. It says the difference between the two variants, if any exists, is smaller than what you can detect with the traffic you have. That is useful: it tells you to stop spending attention on this change and go find a bigger lever.

  • The wrong response is to keep the test running indefinitely hoping the numbers drift somewhere convenient.
  • The second wrong response is to ship the variant anyway because it was ahead.
  • If the data cannot distinguish them, ship whichever is cheaper to maintain and move on.

The trap

Peeking.

Peeking. You watch the dashboard daily, and the moment confidence crosses 95 percent you stop the test and declare a winner. Confidence wanders as data accumulates – it will cross 95 percent by accident on a coin flip given enough looks. Stopping the first time it does turns a 5 percent error rate into something closer to 30 percent.

Decide the sample size before the test starts, run to it, then read the result once. If you must monitor, monitor for disasters, not for wins.

  • The second trap is testing something too small to matter.
  • A button colour that lifts conversion by 0.4 percent relative is untestable and, if it were real, worth almost nothing.
  • Test the offer, the price, the page structure and the traffic source – the things capable of moving a number far enough to be seen.

Frequently asked

Ninety-five percent is the standard threshold. Below it the difference between variants is within the range that random variation would produce anyway.
Almost always sample size. Low baseline conversion rates and small effects need very large numbers of visitors. If the traffic is not there, the test cannot conclude – no amount of waiting fixes an underpowered design.
No. Confidence fluctuates while data accumulates, so stopping at the first crossing dramatically inflates the false positive rate. Fix the sample size in advance and read the result once.