Cohen's Kappa Calculator
The formula
What the beyond-chance agreement tells you
Kappa scores inter-rater reliability after subtracting the agreement random raters would hit by luck.
Calculation: κ = (p₀ − pₑ) ∕ (1 − pₑ). Enter your inputs and every substitution is shown below the result.
Reading the result
80% raw agreement looks great until chance alone explains 68 of it — κ=.38 says the rubric is not holding.
Rules of thumb: κ ≥ .75 reliability for research scoring; ≥ .90 for clinical gates
Limitations and best practice
The prevalence paradox: when one category dominates, kappa can look terrible despite high raw agreement; report both plus a PABAK.
This calculator gives an estimate based on standard formulas; it is not professional financial, medical, legal or engineering advice. Verify anything you plan to act on with a qualified professional and your own records.
Frequently asked questions
Why not just report percent agreement?
What is the kappa paradox?
Two raters or many?
Source & methodology
Cohen (1960), Educational and Psychological Measurement.
How Cohen's Kappa works — and what your result really means
You are here because you need a specific number, and you want to see how it was produced before you rely on it. There are plenty of pages about Cohen's Kappa; almost none of them show the working. This one does: the formula, every input, a fully computed example, and a sensitivity analysis calculated live from this engine’s own defaults.
In plain terms, Cohen's Kappa answers the question "Cohen's kappa?". The measurement matters because it converts a situation that feels complicated — several inputs pulling in different directions — into a single number you can act on. Within Science & Research, Statistics & Data Science is the context where this figure is quoted most often, and professionals treat it as a first-pass screen rather than a verdict.
Like most useful measures, Cohen's Kappa is a summary, and summaries hide as much as they reveal. The companion outputs and the input table are here to reopen what the headline compressed.
What makes a calculation page trustworthy is that its copy is derived from its own arithmetic: here cohen's kappa = 0.19, recomputed every time a field changes, and the sensitivity section ranks the inputs that move the result — Observed agreements first, at ≈29.2 % of the headline for a ten-percent error either way in that input. Read the sections in order and you will be able to reproduce every figure by hand.
Inside the Cohen's Kappa calculation: formula and every input
One formula, published in full: κ = (p₀ − pₑ) ∕ (1 − pₑ). Anything you can read, you can reproduce in a spreadsheet, and that reproducibility is the whole point of printing it rather than hiding it behind a button.
Observed agreements (example value 76) is a numeric input whose unit is fixed by the label; a wrong unit is a wrong answer wearing a plausible disguise. Replace the example with the figure from your own records — the tool recomputes instantly and the printed steps follow your numbers.
Total ratings (example value 190) is a numeric input whose unit is fixed by the label; a wrong unit is a wrong answer wearing a plausible disguise. Replace the example with the figure from your own records — the tool recomputes instantly and the printed steps follow your numbers.
Expected chance agreement (example value 50) is a numeric input whose unit is fixed by the label; a wrong unit is a wrong answer wearing a plausible disguise. The example value is there to demonstrate the mechanics, not to describe your situation — overwrite it.
The default Cohen's Kappa calculation, written out in full
Rather than describe the calculation, here it is end to end — every substitution in order, starting from the inputs above. Starting figures: Observed agreements, Total ratings, Expected chance agreement. The endpoint of that chain is 0.19 of cohen's kappa, sitting in the “Poor — the raw agreement is mostly chance; retune the rubric” band.
- p₀ = 76 ÷ 190 = 0.400
- pₑ (chance) = 50 ÷ 190 = 0.263
- κ = (0.400 − 0.263) ÷ (1 − 0.263) = 0.19
Observed agreement rate — 40.0 %. This is the number experienced users quote when the headline alone would mislead.
Chance agreement rate — 26.3 %. Read next to the headline, this is the figure that shows the shape of the trade-off.
Agreement beyond chance — 13.7 %. This is the number experienced users quote when the headline alone would mislead.
Max possible kappa here — 1.00. Compare it against the main result of 0.19 before drawing conclusions.
Overwrite any input above and every line of this example updates with it — the substitution is live, not a screenshot of somebody else’s numbers.
Scenario table: Cohen's Kappa at ±10 % and ±20 % of the key input
Read this table as the practical error bar on the headline figure. It answers the question people actually have after seeing a single number: how wrong could this be if the main input is off by ten or twenty percent?
| Observed agreements setting | Cohen's kappa |
|---|---|
| 80 % of the example value | 0.08 |
| 90 % of the example value | 0.13 |
| example inputs | 0.19 |
| 110 % of the example value | 0.24 |
| 120 % of the example value | 0.29 |
The corridor runs from 0.08 to 0.29 across that band — which is the range worth planning inside rather than the single 0.19 point estimate. Reproduce it in the form above by editing Observed agreements alone; the substitution steps will follow your figures exactly.
Sensitivity: which inputs change Cohen's Kappa the most
Nobody should have to guess which input matters. The engine was re-run once per field below — ten percent up, ten percent down, everything else frozen — and the results ranked by how far they moved the cohen's kappa. Read it as a map of where Cohen's Kappa is fragile — drawn for the example inputs, which is the map you want when deciding what to measure carefully.
- Observed agreements +10 % → Cohen's Kappa = 0.24 · −10 % → 0.13 (the cohen's kappa shifts ≈29.2 %)
- Expected chance agreement +10 % → Cohen's Kappa = 0.16 · −10 % → 0.21 (the cohen's kappa shifts ≈16.2 %)
- Total ratings +10 % → Cohen's Kappa = 0.16 · −10 % → 0.21 (the cohen's kappa shifts ≈15.7 %)
Observed agreements is the dominant lever here: ten percent either way moves the result by about 29.2 %, from 0.13 to 0.24. That is where a careless figure does the most damage, so check it against a source, and treat Expected chance agreement as the second priority (≈16.2 % per ten percent). The rest of the inputs earn only a rounding-error of attention, and that asymmetry is the practical payoff of measuring rather than asserting.
Reading the result: bands, zones and thresholds
The value above only becomes useful the moment you know what it is being measured against. The verdict attached to the default calculation — “Poor — the raw agreement is mostly chance; retune the rubric” — is one band on a ladder your own figure may sit somewhere else on. How much of that band is luck? On Observed agreements alone ten percent either way covers 0.13 to 0.24. The practical reading rules: compare against your own history before comparing against other people; treat band boundaries as zones, not walls — one point on the wrong side of a threshold rarely changes a real outcome; and when a companion output contradicts the headline, believe the pair.
The comparison worth making is not with somebody else’s Cohen's Kappa but with the version of your own situation measured last quarter — same formula, same conventions, different inputs.
The errors that quietly break a Cohen's Kappa figure
The prevalence paradox: when one category dominates, kappa can look terrible despite high raw agreement; report both plus a PABAK. This calculator gives an estimate based on standard formulas; it is not professional financial, medical, legal or engineering advice. Verify anything you plan to act on with a qualified professional and your own records.
Using stale inputs. Cohen's Kappa is only as current as the numbers behind it — an estimate carried over from a previous season, rate table or report quietly biases every output. Re-measure the inputs first; that single habit fixes more bad Cohen's Kappa figures than any formula change.
This model has its own particular trap: Observed agreements carries ≈29.2 % of the response per ten percent of error, so effort spent elsewhere is largely decorative.
The subtlest error is answering the wrong question precisely. Confirm that cohen's kappa is actually the quantity your decision needs — the arithmetic is flawless either way, which is what makes the mistake invisible.
Before you trust your Cohen's Kappa: a short checklist
- Compare the result with one independent method or published benchmark.
- State, in one sentence, which convention each rate, unit or period in the form is using.
- Re-verify Observed agreements against a source document — ten percent of error there moves the headline by ≈29.2 %.
- Compare the output against one published benchmark or a colleague’s independent estimate.
- Note the date and the inputs beside the number so the figure can be audited later.
- Repeat the ±10 % test on the inputs that matter and see whether the decision — not just the number — still holds.
How to use the Cohen's Kappa tool properly
The tool does the arithmetic; the routine below is what makes the arithmetic trustworthy. Five steps, a minute total — and afterwards you will know not just what the number is, but which assumption produced it.
Step 1 — collect, then type. Everything this tool asks for is listed above (observed agreements, total ratings, expected chance agreement). Pull each figure from something you can point at — paperwork, a dashboard, a label — before touching a field. Guesswork at the input stage cannot be repaired at the interpretation stage. A figure nobody can reproduce is a figure nobody will act on twice.
Step 2 — enter the conservative case first. Resist the flattering scenario. Type the inputs you would defend in front of a sceptical reader, read the Cohen's kappa (0.19 for the example values), and note the verdict band. Suspiciously tidy answers deserve the same scrutiny as untidy ones.
Step 3 — change one input at a time. Nudge a single field by ten percent, watch the headline and its companions move, then undo it. Two or three fields will dominate — the sensitivity table above already ranks them for this engine. If a result feels obviously right, that is the moment to check the inputs hardest.
Step 4 — ask what would have to be true. Take the result you want and work backwards through the inputs until you find the assumption that produces it; then decide honestly whether that assumption is defensible. If a result feels obviously right, that is the moment to check the inputs hardest.
Step 5 — record the context, not just the number. Note today’s date and the inputs alongside the 0.19 output. Six months from now the context is what turns this figure into a trend. A figure with a date beside it becomes evidence; without one it stays an anecdote.
The real-world conversations Cohen's Kappa settles
Science & Research arguments end at arithmetic or they do not end. Cohen's Kappa is the arithmetic: 0.19 on the example inputs, reproducible by anyone who types the same numbers. One decimal place more than your data supports is one decimal place of fiction.
Used properly this is a five-minute filter that saves weeks. Compute Cohen's Kappa, reject what fails, and only then bring in the expensive analysis for the survivors. A figure with a date beside it becomes evidence; without one it stays an anecdote.
Re-running this page costs nothing and catches the slow drifts that never announce themselves. In Science & Research the expensive mistakes are rarely sudden; they are small changes in Observed agreements that nobody recalculated for a year. The point of transparency is that a reader can disagree with a specific line rather than with you.
The questions that come up most about Cohen's Kappa
What affects Cohen's Kappa the most?
The short answer: Observed agreements. The longer answer is the sensitivity table on this page, where every numeric input was probed by ten percent and the cohen's kappa recalculated — Observed agreements moved it furthest, by roughly 29.2 % (0.13 to 0.24).
Which matters more: expected chance agreement or observed agreements?
For Cohen's Kappa, Observed agreements wins the head-to-head (≈29.2 % against ≈16.2 % per ten-percent change). Knowing which input dominates is the difference between careful work and merely slow work.
What is a typical Cohen's Kappa?
On the worked example above Cohen's Kappa comes out at 0.19. Treat that as one point on a distribution rather than a norm: the bands, the companion outputs and the sensitivity ranking tell you how wide the plausible range really is.
How do I calculate Cohen's Kappa by hand?
Three steps: write the formula (κ = (p₀ − pₑ) ∕ (1 − pₑ)), insert your values, and simplify. The worked example section shows exactly that chain for the default inputs; replicating it once on paper teaches you more about Cohen's Kappa than any number of calculator runs.
What happens to Cohen's Kappa if observed agreements changes by 10 %?
The engine was re-run with that field moved by ten percent while everything else stayed fixed: the cohen's kappa goes across 0.13 to 0.24, a swing of roughly 29.2 % around the 0.19 baseline. That ratio is the practical definition of how much this input matters.
Is the Cohen's Kappa result of 0.19 good or bad?
Neither, on its own — the label “Poor — the raw agreement is mostly chance; retune the rubric” is the band this figure falls in for the example inputs, and your own inputs will place you elsewhere on the same ladder. Judge the number against your history and your alternatives, and remember that a ten-percent error in Observed agreements alone moves it by ≈29.2 %.
How many significant figures are reliable in Cohen's Kappa?
About as many as your least precise input. The tool computes at full precision and displays rounded output; report 0.19 to a decimal place your inputs can actually support.
How often should I check Cohen's Kappa?
Whenever an input changes by more than a few percent, and otherwise on the rhythm of your decisions — before a purchase, a review, a renewal. Cohen's Kappa is a snapshot, not a subscription; stale snapshots are the most common silent error in Science & Research planning.
Why not just report percent agreement?
Two coin-flipping raters still agree half the time; kappa credits only what beats that floor.
What is the kappa paradox?
With skewed categories, agreement rises while kappa falls — the chance term assumes the marginals are random.
Does this Cohen's Kappa tool store my numbers?
No. The entire calculation runs in your browser session; nothing entered above is transmitted or saved, which is the correct behaviour for sensitive Science & Research figures and the reason you can re-run this page as often as you like.
Glossary for Cohen's Kappa
- Observed agreement rate
- 40.0 % for the Cohen's Kappa worked case; it sits next to the main result because it carries context the headline leaves out.
- Chance agreement rate
- 26.3 % on the example inputs — a companion figure printed beside the headline of Cohen's Kappa so that no result is read in isolation.
- Agreement beyond chance
- 13.7 % in the default Cohen's Kappa calculation — one of the secondary figures an experienced reader checks before trusting the primary number.
- Cohen's kappa
- The primary output of this tool: 0.19 for the default inputs, computed exactly from κ = (p₀ − pₑ) ∕ (1 − pₑ).
- Worked example
- The calculation written out in full so that nothing on this page has to be taken on trust.
- Verdict band
- A named zone on the range of plausible results; the example inputs land in “Poor — the raw agreement is mostly chance; retune the rubric”, and your inputs may not.
- Sensitivity (±10 %)
- How far the Cohen's Kappa moves when a single input is nudged ten percent either way; the ranked list on this page is computed live from this engine.
Where the Cohen's Kappa model comes from
The model is published arithmetic implemented exactly as written: κ = (p₀ − pₑ) ∕ (1 − pₑ). Source of record: Cohen (1960), Educational and Psychological Measurement. Any other implementation of this model should return the same figure for the same inputs. If one does not, the difference is in its assumptions — which this page prints and that one probably does not.
Treat the output as a precise answer to the question you typed: exact arithmetic, transparent formula, published source, no hidden rounding. The design constraint was set by usefulness rather than by comfort: an estimate you can interrogate beats an answer you have to trust.
After the number: what to do with it
Save the number together with its inputs and today’s date; a Cohen's Kappa figure without context is unusable the moment an input changes.
This page was written for people who need one trustworthy number before their next decision, and the promise is that nothing above required trust. Bookmark the tool, re-run it whenever an input moves by more than a few percent, and keep the date with the number.