A/B Test & Statistical Sample Size Calculator
The formula
The Deadly "Peeking Problem" in A/B Testing
The most common mistake product teams make is early peeking: checking results daily and stopping the test the moment a statistical software dashboard flashes "95% Significant!". Under classical frequentist statistics, checking results continuously inflates your real false-positive rate from 5% to over 30% due to temporary random-walk fluctuations.
You must compute your sample size upfront with this calculator, commit to the sample size, and evaluate the primary metric only once the target sample has been fully collected.
Why 1–2 Full Weeks of Duration is Non-Negotiable
Even if a high-traffic website can collect 100,000 visitors in 36 hours, you should never conclude a test in less than 7 days. User behavior differs drastically on Sunday night versus Tuesday morning. Running an experiment for full 7-day or 14-day cycles normalizes for weekly conversion seasonality and payday effects.
Frequently asked questions
What is the difference between One-Tailed and Two-Tailed tests?
What if I cannot reach the required sample size in a reasonable time?
Does 95% significance mean my variant has a 95% chance of being better?
Source & methodology
Evan Miller (Sample Size Formula for Two Proportions); Kohavi, Tang & Xu (Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press).
A/B Test & Statistical Sample Size: the number, the method and the caveats
This page was written for one job: turning your own inputs into a defensible figure, with the working shown. Searchers land on a A/B Test & Statistical Sample Size page for three reasons — to plan, to compare options, and to check that somebody else’s arithmetic is honest. This tool serves all three, and everything after the calculator explains the model instead of hiding it.
Like most useful measures, A/B Test & Statistical Sample Size is a summary, and summaries hide as much as they reveal. The companion outputs and the input table are here to reopen what the headline compressed.
It helps to separate the mechanics from the interpretation. The mechanics are fixed: n = [ Z_α·√(2·p̄·(1−p̄)) + Z_β·√(p₁·(1−p₁) + p₂·(1−p₂)) ]² / (p₂ − p₁)² | Duration = 2n / Daily Traffic. Anyone, anywhere, entering the same inputs will get the same 31,789 visitors / arm. The interpretation is where experience counts: whether 31,789 visitors / arm is good, normal or alarming depends on comparisons covered below, and that is the part most articles about A/B Test & Statistical Sample Size never explain.
What makes a calculation page trustworthy is that its copy is derived from its own arithmetic: here sample Size Required per Variant = 31,789 visitors / arm, recomputed every time a field changes, and the sensitivity section ranks the inputs that move the result — Minimum Detectable Effect — MDE first, at ≈22.8 % of the headline for a ten-percent error either way in that input. Read the sections in order and you will be able to reproduce every figure by hand.
How A/B Test & Statistical Sample Size is computed, line by line
One formula, published in full: n = [ Z_α·√(2·p̄·(1−p̄)) + Z_β·√(p₁·(1−p₁) + p₂·(1−p₂)) ]² / (p₂ − p₁)² | Duration = 2n / Daily Traffic. Anything you can read, you can reproduce in a spreadsheet, and that reproducibility is the whole point of printing it rather than hiding it behind a button.
Baseline conversion rate (%) (example value 4) is a measured number in the unit named by the label; the formula does not convert for you. Rates are the most mis-transcribed input on any page like this; copy it from the document rather than retyping from memory.
Minimum Detectable Effect — MDE (% relative lift) (example value 12) is a quantity rather than a category — precision here is cheap and worth having. Check the convention first: if the source says “6.5 % per year”, type 6.5 and let the formula do the rest.
Statistical significance level (α) / Confidence (example value 95% Confidence (α = 0.05 — Tech & Software Standard)) is a choice rather than a quantity — pick the option that matches your situation, because the calculation path branches here. If you are unsure what belongs in this field, look at the printed steps: the substitution shows exactly how the engine uses it.
Statistical power (1 − β) (example value 80% Power (Industry Standard — 20% risk of missing real effect)) is a categorical input — the option you choose decides which numbers the formula uses. The example value is there to demonstrate the mechanics, not to describe your situation — overwrite it.
Hypothesis testing tails (example value Two-Tailed (Standard — tests for both positive lift and negative regression)) is a choice rather than a quantity — pick the option that matches your situation, because the calculation path branches here. The example value is there to demonstrate the mechanics, not to describe your situation — overwrite it.
Available daily traffic / impressions (total across all variants) (example value 8,000) is a quantity rather than a category — precision here is cheap and worth having. Replace the example with the figure from your own records — the tool recomputes instantly and the printed steps follow your numbers.
What each “Statistical significance level (α) / Confidence” choice actually does: 99% Confidence (α = 0.01 — Medical & Clinical Trials) → 47,302 visitors / arm · 95% Confidence (α = 0.05 — Tech & Software Standard) → 31,789 visitors / arm · 90% Confidence (α = 0.10 — Fast Exploratory Testing) → 25,041 visitors / arm These figures are computed from this engine, not estimated — switch the dropdown above and you will land on exactly these numbers.
What each “Statistical power (1 − β)” choice actually does: 80% Power (Industry Standard — 20% risk of missing real effect) → 31,789 visitors / arm · 90% Power (High Rigor — 10% risk of Type II error) → 42,556 visitors / arm · 95% Power (Extremely High Sensitivity) → 52,630 visitors / arm These figures are computed from this engine, not estimated — switch the dropdown above and you will land on exactly these numbers.
A fully worked A/B Test & Statistical Sample Size example
The tool prints its own working; this is the same arithmetic on the example values, reproduced so you can follow it by hand. Starting figures: Baseline conversion rate (%), Minimum Detectable Effect — MDE (% relative lift), Statistical significance level (α) / Confidence, Statistical power (1 − β). The computed sample Size Required per Variant comes out at 31,789 visitors / arm, which the interpretation band labels “🟢 Optimal Test Duration: Can be executed cleanly in 8 days (1.1 weeks) with reliable results.”.
- Baseline Control Rate $p_1 = 3.5% = 0.0350$, Target Variant Rate $p_2 = 0.0350 imes (1 + 0.12) = 0.0392$ (3.92%)
- Pooled Proportion $\bar{p} = (p_1 + p_2) ÷ 2 = 0.0371$, Absolute Delta $\Delta = 0.0042$
- Critical Z-Scores: $Z_{\alpha} = 1.96$ (for 95% confidence), $Z_{\beta} = 0.84$ (for 80% power)
- Numerator = $\left[1.96\sqrt{2(0.037)(0.963)} + 0.84\sqrt{0.035(0.965) + 0.039(0.961)}\right]^2$
- Required Sample per Variant = Numerator ÷ $(0.0042)^2 = $ 31,789 users
- Total Traffic Required = $31,789 \times 2 = $ 63,578 users
Total Sample Size (Control + Variant) — 63,578 visitors. If your inputs change, this is usually the first companion figure to move.
Estimated Experiment Duration — 8 Days. If your inputs change, this is usually the first companion figure to move.
Target Variant Conversion Rate — 3.92%. If your inputs change, this is usually the first companion figure to move.
Absolute Conversion Lift Required — 0.42 percentage pts. This is the number experienced users quote when the headline alone would mislead.
Statistical Confidence Level — 95%. This is the number experienced users quote when the headline alone would mislead.
Overwrite any input above and every line of this example updates with it — the substitution is live, not a screenshot of somebody else’s numbers.
How far A/B Test & Statistical Sample Size can reasonably move
Read this table as the practical error bar on the headline figure. It answers the question people actually have after seeing a single number: how wrong could this be if the main input is off by ten or twenty percent?
| Minimum Detectable Effect — MDE setting | Sample Size Required per Variant |
|---|---|
| 80 % of the example value | 49,131 visitors / arm |
| 90 % of the example value | 39,033 visitors / arm |
| example inputs | 31,789 visitors / arm |
| 110 % of the example value | 26,415 visitors / arm |
| 120 % of the example value | 22,316 visitors / arm |
The corridor runs from 22,316 visitors / arm to 49,131 visitors / arm across that band — which is the range worth planning inside rather than the single 31,789 visitors / arm point estimate. Reproduce it in the form above by editing Minimum Detectable Effect — MDE alone; the substitution steps will follow your figures exactly.
Where A/B Test & Statistical Sample Size is fragile — and where it is not
This list was computed for this specific engine just now: each numeric input was nudged ±10 % while every other field held its example value, and the sample Size Required per Variant was recalculated from scratch. Read it as a map of where A/B Test & Statistical Sample Size is fragile — drawn for the example inputs, which is the map you want when deciding what to measure carefully.
- Minimum Detectable Effect — MDE +10 % → A/B Test & Statistical Sample Size = 26,415 visitors / arm · −10 % → 39,033 visitors / arm (the sample Size Required per Variant shifts ≈22.8 %)
- Nudge Baseline conversion rate by ±10 % and A/B Test & Statistical Sample Size lands between 28,788 visitors / arm and 35,458 visitors / arm (sensitivity ≈11.5 %)
Minimum Detectable Effect — MDE is the dominant lever here: ten percent either way moves the result by about 22.8 %, from 26,415 visitors / arm to 39,033 visitors / arm. That is where a careless figure does the most damage, so check it against a source, and treat Baseline conversion rate as the second priority (≈11.5 % per ten percent). The rest of the inputs earn only a rounding-error of attention, and that asymmetry is the practical payoff of measuring rather than asserting.
What a A/B Test & Statistical Sample Size of 31,789 visitors / arm means in practice
The value above only becomes useful the moment you know what it is being measured against. The verdict attached to the default calculation — “🟢 Optimal Test Duration: Can be executed cleanly in 8 days (1.1 weeks) with reliable results.” — is one band on a ladder your own figure may sit somewhere else on. The sensitivity list shows how far a modest input error moves you along that ladder: for Minimum Detectable Effect — MDE, ±10 % covers 26,415 visitors / arm to 39,033 visitors / arm. The practical reading rules: ask what would have to change to move you one band. If the answer is “a small, plausible input difference”, the band itself is not the finding — the uncertainty is.
Where A/B Test & Statistical Sample Size sits next to similar measures: this tool answers “how much is the A/B Test & Statistical Sample Size”, while the related calculators answer “what do I do about it”. Reading the result alone tends to over-weight a single snapshot; pairing it with the companion tools above turns the number into a decision.
Why A/B Test & Statistical Sample Size results go wrong — and how to prevent it
Using stale inputs. A/B Test & Statistical Sample Size is only as current as the numbers behind it — an estimate carried over from a previous season, rate table or report quietly biases every output. Re-measure the inputs first; that single habit fixes more bad A/B Test & Statistical Sample Size figures than any formula change.
Using stale inputs. A/B Test & Statistical Sample Size is only as current as the numbers behind it — an estimate carried over from a previous season, rate table or report quietly biases every output. Re-measure the inputs first; that single habit fixes more bad A/B Test & Statistical Sample Size figures than any formula change.
This model has its own particular trap: Minimum Detectable Effect — MDE carries ≈22.8 % of the response per ten percent of error, so effort spent elsewhere is largely decorative.
Another is comparing across contexts: a A/B Test & Statistical Sample Size computed with one set of conventions should never be lined up against a figure computed with another. Check the inputs behind both before ranking them.
A quick check for any A/B Test & Statistical Sample Size figure
- State, in one sentence, which convention each rate, unit or period in the form is using.
- Compare the result with one independent method or published benchmark.
- Put Minimum Detectable Effect — MDE under a magnifier; the sensitivity table above is the reason why.
- Note the date and the inputs beside the number so the figure can be audited later.
- Recompute once with the dominant input deliberately set low, and see whether the decision changes.
- Repeat the ±10 % test on the inputs that matter and see whether the decision — not just the number — still holds.
Step by step: getting a trustworthy A/B Test & Statistical Sample Size out of this tool
The tool does the arithmetic; the routine below is what makes the arithmetic trustworthy. Five steps, a minute total — and afterwards you will know not just what the number is, but which assumption produced it.
Step 1 — start from evidence. The inputs behind A/B Test & Statistical Sample Size are baseline conversion rate (%), minimum detectable effect — mde (% relative lift), statistical significance level (α) / confidence, statistical power (1 − β), hypothesis testing tails, available daily traffic / impressions (total across all variants). Write them down with their source next to each one; if you cannot name where a figure came from, that is the input most likely to be wrong. When two sources disagree, it is almost never the arithmetic that differs.
Step 2 — enter the conservative case first. Resist the flattering scenario. Type the inputs you would defend in front of a sceptical reader, read the Sample Size Required per Variant (31,789 visitors / arm for the example values), and note the verdict band. The point of transparency is that a reader can disagree with a specific line rather than with you.
Step 3 — measure what deserves measuring. The ranked levers above tell you where a careful figure pays off. Everything else can stay approximate, which is a licence to stop hunting for data you do not need. Estimates improve most when the worst case is written down in ink.
Step 4 — ask what would have to be true. Take the result you want and work backwards through the inputs until you find the assumption that produces it; then decide honestly whether that assumption is defensible. Half of a good calculation is knowing which input you would hate to be wrong about.
Step 5 — save the story with the figure. A number without its inputs and its date is unusable within a quarter. Record 31,789 visitors / arm, the assumptions behind it, and today’s date; your future self will be auditing a stranger’s arithmetic. The input you reach for first is usually the one you measured longest ago.
Where A/B Test & Statistical Sample Size matters in practice
Science & Research arguments end at arithmetic or they do not end. A/B Test & Statistical Sample Size is the arithmetic: 31,789 visitors / arm on the example inputs, reproducible by anyone who types the same numbers. If the answer changes your behaviour, it deserves a second run with different assumptions.
Professionals in Statistics & Data Science use this style of calculation as a screening filter before spending real money or effort: a quick A/B Test & Statistical Sample Size check eliminates weak options in minutes, and the survivors earn the deeper, slower analysis. The inputs are the part of the work that nobody else can do for you.
There is a personal-administration use people overlook: periodic self-checking. A/B Test & Statistical Sample Size quietly drifts as life changes — rates move, bodies change, costs compound — and re-running it a few times a year catches drift while correction is still cheap. When in doubt, prefer the boring input: the interesting one is usually the one you misread.
People also ask about A/B Test & Statistical Sample Size
What affects A/B Test & Statistical Sample Size the most?
Measured on this engine, Minimum Detectable Effect — MDE is the dominant lever: moving it ±10 % shifts the sample Size Required per Variant across 26,415 visitors / arm to 39,033 visitors / arm — about 22.8 % of the headline, with Baseline conversion rate next at ≈11.5 %. The ranked list in the sensitivity section is computed live, not asserted from a template.
Which matters more: baseline conversion rate or minimum detectable effect — mde?
Minimum Detectable Effect — MDE, by a factor of about 2.0. Concretely: ten percent on Minimum Detectable Effect — MDE is worth ≈22.8 % of A/B Test & Statistical Sample Size, the same nudge on Baseline conversion rate only ≈11.5 %. That ranking is re-computed for each page from that page’s own defaults.
How much does the “Statistical significance level (α) / Confidence” choice change the answer?
The spread is not cosmetic: this choice alone takes A/B Test & Statistical Sample Size from 25,041 visitors / arm to 47,302 visitors / arm with every other input frozen, which is more influence than most of the numeric fields on this page can claim.
What is a typical A/B Test & Statistical Sample Size?
With the example inputs on this page the result is 31,789 visitors / arm, labelled “🟢 Optimal Test Duration: Can be executed cleanly in 8 days (1.1 weeks) with reliable results.”. “Typical” varies with situation, which is why every result here ships with its verdict band and its sensitivity list: they turn one figure into a range you can locate yourself inside.
How do I calculate A/B Test & Statistical Sample Size by hand?
n = [ Z_α·√(2·p̄·(1−p̄)) + Z_β·√(p₁·(1−p₁) + p₂·(1−p₂)) ]² / (p₂ − p₁)² | Duration = 2n / Daily Traffic is the entire method. Work the inputs in that order — Baseline Control Rate $p_1 = 3.5% = 0.0350$, Target Variant Rate $p_2 = 0.0350 imes (1 + 0.12) = 0.0392$ (3.92%) — and you will reach 31,789 visitors / arm on the default figures, the same number this tool prints.
What happens to A/B Test & Statistical Sample Size if minimum detectable effect — mde changes by 10 %?
Try it in the form above and the printed substitution will follow you; the pre-computed answer for the defaults is 26,415 visitors / arm to 39,033 visitors / arm across the ten-percent probe, i.e. ≈22.8 % sensitivity on the sample Size Required per Variant.
Is the A/B Test & Statistical Sample Size result of 31,789 visitors / arm good or bad?
Neither, on its own — the label “🟢 Optimal Test Duration: Can be executed cleanly in 8 days (1.1 weeks) with reliable results.” is the band this figure falls in for the example inputs, and your own inputs will place you elsewhere on the same ladder. Judge the number against your history and your alternatives, and remember that a ten-percent error in Minimum Detectable Effect — MDE alone moves it by ≈22.8 %.
How do I calculate A/B Test & Statistical Sample Size manually?
Write down the inputs, then follow the printed steps in order — n = [ Z_α·√(2·p̄·(1−p̄)) + Z_β·√(p₁·(1−p₁) + p₂·(1−p₂)) ]² / (p₂ − p₁)² | Duration = 2n / Daily Traffic. A spreadsheet replicates it with two or three cells. The manual route is worth doing once to build intuition; after that the calculator saves time and eliminates transcription slips.
How many significant figures are reliable in A/B Test & Statistical Sample Size?
About as many as your least precise input. The tool computes at full precision and displays rounded output; report 31,789 visitors / arm to a decimal place your inputs can actually support.
What is the difference between One-Tailed and Two-Tailed tests?
A Two-Tailed test checks whether your variant is either statistically significantly better OR worse than the control. A One-Tailed test only checks if it is better. Unless there is zero business risk in releasing a broken feature, always use Two-Tailed testing.
What if I cannot reach the required sample size in a reasonable time?
You have three engineering levers: (1) Test a more bold, radical design change that targets a larger MDE (e.g. 20% instead of 5%), (2) Move your test higher up the conversion funnel where visitor volume is higher (e.g. signup button clicks instead of final checkout purchases), or (3) Adopt Bayesian testing frameworks.
Is my data private on this page?
Yes — computation happens locally in your browser, with no account, no tracking of your inputs and nothing sent to a server. The printed formula is the whole story of what happens to your figures.
The vocabulary of A/B Test & Statistical Sample Size
- Total Sample Size (Control + Variant)
- 63,578 visitors on the example inputs — a companion figure printed beside the headline of A/B Test & Statistical Sample Size so that no result is read in isolation.
- Estimated Experiment Duration
- 8 Days for the A/B Test & Statistical Sample Size worked case; it sits next to the main result because it carries context the headline leaves out.
- Target Variant Conversion Rate
- 3.92% in the default A/B Test & Statistical Sample Size calculation — one of the secondary figures an experienced reader checks before trusting the primary number.
- Sample Size Required per Variant
- The primary output of this tool: 31,789 visitors / arm for the default inputs, computed exactly from n = [ Z_α·√(2·p̄·(1−p̄)) + Z_β·√(p₁·(1−p₁) + p₂·(1−p₂)) ]² / (p₂ − p₁)² | Duration = 2n / Daily Traffic.
- Worked example
- The step-by-step substitution printed above, so the arithmetic can be re-derived by hand or in a spreadsheet.
- Verdict band
- The interpretation attached to a result (“🟢 Optimal Test Duration: Can be executed cleanly in 8 days (1.1 weeks) with reliable results.” on these inputs), which turns a bare number into a position on a scale.
- Sensitivity (±10 %)
- How far the A/B Test & Statistical Sample Size moves when a single input is nudged ten percent either way; the ranked list on this page is computed live from this engine.
Methodology, sources and limits
The model is published arithmetic implemented exactly as written: n = [ Z_α·√(2·p̄·(1−p̄)) + Z_β·√(p₁·(1−p₁) + p₂·(1−p₂)) ]² / (p₂ − p₁)² | Duration = 2n / Daily Traffic. Source of record: Evan Miller (Sample Size Formula for Two Proportions); Kohavi, Tang & Xu (Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press). Where a professional version of this calculation differs, it differs in inputs and jurisdiction-specific coefficients, not in the algebra — which is why re-checking your inputs beats hunting for a second calculator.
Nothing here replaces a qualified professional, and nothing here is approximate either — the arithmetic is exact, and the honesty is in the inputs you supply. The sensitivity table above is part of that promise rather than a bonus: it lets you see how much of the result depends on how carefully you measured.
What to do with your A/B Test & Statistical Sample Size next
Save the number together with its inputs and today’s date; a A/B Test & Statistical Sample Size figure without context is unusable the moment an input changes.
This page was written for people who need one trustworthy number before their next decision, and the promise is that nothing above required trust. Bookmark the tool, re-run it whenever an input moves by more than a few percent, and keep the date with the number.