A/B Test Significance Calculator
The formula
What the a/b test significance tells you
The two-proportion z-test for a control-versus-variant comparison, reporting lift, the z-score, and the p-value against the 95% bar.
Calculation: Two-proportion z-test on the observed rates. Enter your inputs and every substitution is shown below the result.
Reading the result
Significance is a decision tool, not a trophy: a significant 2% lift on the checkout may beat a significant 40% on a footer link — weight by value.
Rules of thumb: p < 0.05 is the convention; 0.005–0.01 for high-stakes irreversible decisions
Limitations and best practice
Peeking kills the math: checking at day three and stopping at first-under-p inflates false positives to 20–30%; pre-commit to the sample and analyse once.
This calculator gives an estimate based on standard formulas; it is not professional financial, medical, legal or engineering advice. Verify anything you plan to act on with a qualified professional and your own records.
Frequently asked questions
One-sided or two-sided test?
What if traffic splits are uneven?
Source & methodology
Standard two-proportion hypothesis test.
A/B Test Significance: what this tool answers
This page was written for one job: turning your own inputs into a defensible figure, with the working shown. A/B Test Significance is the kind of number people argue about from memory. This page ends that argument with arithmetic you can watch happen: enter your figures, get Lift, variant vs control in real time, and see every substitution printed below.
Like most useful measures, A/B Test Significance is a summary, and summaries hide as much as they reveal. The companion outputs and the input table are here to reopen what the headline compressed.
It helps to separate the mechanics from the interpretation. The mechanics are fixed: Two-proportion z-test on the observed rates. Anyone, anywhere, entering the same inputs will get the same 35.3 %. The interpretation is where experience counts: whether 35.3 % is good, normal or alarming depends on comparisons covered below, and that is the part most articles about A/B Test Significance never explain.
What makes a calculation page trustworthy is that its copy is derived from its own arithmetic: here lift, variant vs control = 35.3 %, recomputed every time a field changes, and the sensitivity section ranks the inputs that move the result — Control conversions first, at ≈42.6 % of the headline for a ten-percent error either way in that input. Work down the page and by the end you could re-derive the result in a spreadsheet from memory.
How A/B Test Significance is computed, line by line
The engine implements a single stated model: Two-proportion z-test on the observed rates. There is no hidden coefficient and no login-gated precision — the formula printed is the formula executed, which is also what makes the result citable.
Control conversions (example value 317) is a measured number in the unit named by the label; the formula does not convert for you. If you are unsure what belongs in this field, look at the printed steps: the substitution shows exactly how the engine uses it.
Control visitors (example value 12,696) is a measured number in the unit named by the label; the formula does not convert for you. This input feeds the formula directly; a defensible figure here is worth more than false precision anywhere else.
Variant conversions (example value 395) is a quantity rather than a category — precision here is cheap and worth having. This input feeds the formula directly; a defensible figure here is worth more than false precision anywhere else.
Variant visitors (example value 11,691) is a plain measured quantity — the unit shown in the label is the unit the formula expects. If you are unsure what belongs in this field, look at the printed steps: the substitution shows exactly how the engine uses it.
A/B Test Significance on the example inputs: the complete substitution
Rather than describe the calculation, here it is end to end — every substitution in order, starting from the inputs above. Starting figures: Control conversions, Control visitors, Variant conversions, Variant visitors. The endpoint of that chain is 35.3 % of lift, variant vs control, sitting in the “Statistically significant (p = 0.000) — ship it” band.
- Rates: 2.50% vs 3.38%
- z = (0.0338 − 0.0250) ÷ 0.00216 = 4.09, two-sided p = 0.000
z-score — 4.09. Compare it against the main result of 35.3 % before drawing conclusions.
p-value — 0.000. Compare it against the main result of 35.3 % before drawing conclusions.
Control rate — 2.50 %. Read next to the headline, this is the figure that shows the shape of the trade-off.
Re-running with your own figures is instant; re-running by hand once is the best way to know the tool is telling you the truth.
The A/B Test Significance corridor: your result across a realistic band
Read this table as the practical error bar on the headline figure. It answers the question people actually have after seeing a single number: how wrong could this be if the main input is off by ten or twenty percent?
| Control conversions setting | Lift, variant vs control |
|---|---|
| 80 % of the example value | 69.1 % |
| 90 % of the example value | 50.4 % |
| example inputs | 35.3 % |
| 110 % of the example value | 23.0 % |
| 120 % of the example value | 12.8 % |
Across 5 settings of Control conversions, the result spans 12.8 % to 69.1 %. A decision that only works at one row of this table is a decision waiting for an input to move. Reproduce it in the form above by editing Control conversions alone; the substitution steps will follow your figures exactly.
What actually moves your A/B Test Significance
This list was computed for this specific engine just now: each numeric input was nudged ±10 % while every other field held its example value, and the lift, variant vs control was recalculated from scratch. It is the fastest honest answer to “what matters most in A/B Test Significance?” — and it is specific to these defaults, because sensitivity moves with the starting point.
- Nudge Control conversions by ±10 % and A/B Test Significance lands between 23.0 % and 50.4 % (sensitivity ≈42.6 %)
- Variant visitors +10 % → A/B Test Significance = 23.0 % · −10 % → 50.4 % (the lift, variant vs control shifts ≈42.6 %)
- Variant conversions: ten percent up gives 48.8 %, ten percent down gives 21.8 % — ≈38.3 % of the lift, variant vs control either way
- Nudge Control visitors by ±10 % and A/B Test Significance lands between 21.8 % and 48.8 % (sensitivity ≈38.3 %)
Control conversions is the dominant lever here: ten percent either way moves the result by about 42.6 %, from 23.0 % to 50.4 %. Spend your verification effort there first, and Variant visitors is effectively tied with it (≈42.6 % for the same nudge), so verify the pair together. Everything below that can stay approximate, which is a licence to stop chasing data that would not change the decision.
What a A/B Test Significance of 35.3 % means in practice
Nobody acts on a bare figure — they act on a figure with a comparison attached. The default case lands in the “Statistically significant (p = 0.000) — ship it” band, and bands are deliberately wide because real situations overlap. The sensitivity list shows how far a modest input error moves you along that ladder: for Control conversions, ±10 % covers 23.0 % to 50.4 %. The practical reading rules: read the band as a starting hypothesis, then test it: change the two dominant inputs by ten percent and see whether the verdict survives. Verdicts that survive stress are the ones worth acting on.
Set A/B Test Significance beside its neighbours and it usually plays a specific role: this figure establishes the size of something, and the neighbouring tools establish whether that size is acceptable. Skipping the first step makes the second one arbitrary.
How people get A/B Test Significance wrong
Peeking kills the math: checking at day three and stopping at first-under-p inflates false positives to 20–30%; pre-commit to the sample and analyse once. This calculator gives an estimate based on standard formulas; it is not professional financial, medical, legal or engineering advice. Verify anything you plan to act on with a qualified professional and your own records.
Reading A/B Test Significance from a single month. Business figures have seasonality and noise; compare across a full cycle before believing the trend.
This model has its own particular trap: Control conversions carries ≈42.6 % of the response per ten percent of error, so effort spent elsewhere is largely decorative.
The subtlest error is answering the wrong question precisely. Confirm that lift, variant vs control is actually the quantity your decision needs — the arithmetic is flawless either way, which is what makes the mistake invisible.
A/B Test Significance verification checklist
- Verify each input against its original source document, not memory.
- Record the date and the inputs alongside the number so future-you knows what was assumed.
- Put Control conversions under a magnifier; the sensitivity table above is the reason why.
- Note the date and the inputs beside the number so the figure can be audited later.
- Check that every field the formula needs actually arrived from a document, not from recollection.
- Run the ±10 % stress on the top two levers and check that your conclusion survives both.
Step by step: getting a trustworthy A/B Test Significance out of this tool
Using a calculator well sounds trivial, but the difference between a decision-grade figure and a throwaway one lives entirely in the routine. The routine for A/B Test Significance takes a minute to learn and applies to every input above.
Step 1 — gather before you type. A/B Test Significance depends on control conversions, control visitors, variant conversions, variant visitors. Collect them from a source document — a statement, a report, an invoice, a log — rather than from memory, because recalled inputs are the single largest cause of wrong results. A result worth acting on survives having its two shakiest inputs halved.
Step 2 — start from the case you can defend. Enter pessimistic-but-honest figures first; the optimistic run is a ceiling, not a plan. A conclusion that holds on conservative inputs holds everywhere. Estimates improve most when the worst case is written down in ink.
Step 3 — isolate your variables. Change one field, observe, revert. The live sensitivity list does the ranking for the default inputs; verify it against your own, because dominance shifts when the starting point shifts. If the verdict changes with a ten-percent nudge, report the nudge alongside the number.
Step 4 — run the scenario trio. Expected, pessimistic, optimistic. The gap between pessimistic and expected is the number to plan against; the optimistic result is useful only as a ceiling. The inputs are the part of the work that nobody else can do for you.
Step 5 — schedule the next run. In Business & Entrepreneurship the inputs drift quietly. Put a reminder a quarter out, re-enter the same fields, and read the change rather than the level. The test of a good estimate is not that it feels right but that it changes what you do.
Who actually uses A/B Test Significance — and when
Most Marketing & Growth disagreements are arithmetic disagreements in disguise. When Lift, variant vs control of 35.3 % sits on the table with its working shown, the conversation stops being about who sounds more confident and starts being about which input is wrong. The point of transparency is that a reader can disagree with a specific line rather than with you.
Professionals in Marketing & Growth use this style of calculation as a screening filter before spending real money or effort: a quick A/B Test Significance check eliminates weak options in minutes, and the survivors earn the deeper, slower analysis. If a result feels obviously right, that is the moment to check the inputs hardest.
Nothing here is stored or transmitted, so the habit of checking is free: open the page, re-enter current figures, compare with the number you saved last time. Re-running a number costs seconds; retracting a decision costs considerably more.
The questions that come up most about A/B Test Significance
What affects A/B Test Significance the most?
A/B Test Significance is driven above all by Control conversions. Perturbing only that field by ten percent either way produced 23.0 % against the 35.3 % baseline — a ≈42.6 % response, while Variant visitors manages ≈42.6 %.
Do variant visitors and control conversions matter equally?
Effectively. Measured on this engine they land at ≈42.6 % and ≈42.6 % for the same ten-percent nudge, so the honest advice is to get both right; chasing precision in only one of them buys almost nothing.
What is a typical A/B Test Significance?
With the example inputs on this page the result is 35.3 %, labelled “Statistically significant (p = 0.000) — ship it”. “Typical” varies with situation, which is why every result here ships with its verdict band and its sensitivity list: they turn one figure into a range you can locate yourself inside.
How do I calculate A/B Test Significance by hand?
Two-proportion z-test on the observed rates is the entire method. Work the inputs in that order — Rates: 2.50% vs 3.38% — and you will reach 35.3 % on the default figures, the same number this tool prints.
What happens to A/B Test Significance if control conversions changes by 10 %?
Try it in the form above and the printed substitution will follow you; the pre-computed answer for the defaults is 23.0 % to 50.4 % across the ten-percent probe, i.e. ≈42.6 % sensitivity on the lift, variant vs control.
Is the A/B Test Significance result of 35.3 % good or bad?
Neither, on its own — the label “Statistically significant (p = 0.000) — ship it” is the band this figure falls in for the example inputs, and your own inputs will place you elsewhere on the same ladder. Judge the number against your history and your alternatives, and remember that a ten-percent error in Control conversions alone moves it by ≈42.6 %.
Should I use A/B Test Significance before launching?
Yes — a A/B Test Significance run on honest inputs before spending is the cheapest consulting an entrepreneur can buy. Re-run at every pricing change; the number moves more often than founders expect.
How often should I check A/B Test Significance?
Whenever an input changes by more than a few percent, and otherwise on the rhythm of your decisions — before a purchase, a review, a renewal. A/B Test Significance is a snapshot, not a subscription; stale snapshots are the most common silent error in Business & Entrepreneurship planning.
One-sided or two-sided test?
Two-sided by default — you genuinely might have made it worse; one-sided halves the sample but quietly doubles regret when you ship a decline.
What if traffic splits are uneven?
The pooled-se formula above already handles unequal n; extreme imbalance (90/10) loses power, so rebalance for the next test.
Does this A/B Test Significance tool store my numbers?
No. The entire calculation runs in your browser session; nothing entered above is transmitted or saved, which is the correct behaviour for sensitive Business & Entrepreneurship figures and the reason you can re-run this page as often as you like.
Glossary for A/B Test Significance
- z-score
- 4.09 in the default A/B Test Significance calculation — one of the secondary figures an experienced reader checks before trusting the primary number.
- p-value
- 0.000 in the default A/B Test Significance calculation — one of the secondary figures an experienced reader checks before trusting the primary number.
- Control rate
- 2.50 % on the example inputs — a companion figure printed beside the headline of A/B Test Significance so that no result is read in isolation.
- Lift, variant vs control
- The primary output of this tool: 35.3 % for the default inputs, computed exactly from Two-proportion z-test on the observed rates.
- Worked example
- Every substitution in order, from the example inputs to 35.3 % — the audit trail for the headline figure.
- Verdict band
- The plain-language label — here “Statistically significant (p = 0.000) — ship it” — mapping the computed figure onto experience-based ranges for Business & Entrepreneurship.
- Sensitivity (±10 %)
- How far the A/B Test Significance moves when a single input is nudged ten percent either way; the ranked list on this page is computed live from this engine.
Where the A/B Test Significance model comes from
The model is published arithmetic implemented exactly as written: Two-proportion z-test on the observed rates. Source of record: Standard two-proportion hypothesis test. Where a professional version of this calculation differs, it differs in inputs and jurisdiction-specific coefficients, not in the algebra — which is why re-checking your inputs beats hunting for a second calculator.
This page is an estimation aid, not professional advice. The design rule is simple: two people entering identical figures must get identical answers. Nothing is smoothed, weighted or adjusted behind the scenes; where judgement enters, this page says so and names the input responsible.
After the number: what to do with it
Save the number together with its inputs and today’s date; a A/B Test Significance figure without context is unusable the moment an input changes.
This page was written for people who need one trustworthy number before their next decision, and the promise is that nothing above required trust. Bookmark the tool, re-run it whenever an input moves by more than a few percent, and keep the date with the number.