StatReason

Weighted Kappa Calculator

Quantify agreement between two raters when the categories have a natural order, such as mild, moderate, and severe, giving partial credit when raters are close but not identical.

What this answers

This calculator answers "how much do two raters agree, giving credit for being close even when they are not exactly identical?" a more forgiving and often more realistic question for ordered categories than plain Cohen's kappa, which treats every disagreement as equally bad.

How it is calculated

Every pair of categories gets a disagreement weight based on how far apart they are on the ordinal scale, normalized so the two most extreme categories get a weight of 1. Linear weighting scales that penalty proportionally to the distance; quadratic weighting squares it, punishing large disagreements much more heavily than small ones. Weighted kappa then compares the weighted observed disagreement to the weighted disagreement expected by chance, the same logic as ordinary kappa but applied through these distance-aware weights instead of a simple match/no-match rule.

Worked example

For two raters who rate the same five items identically as 1, 2, 3, 1, and 2, every pair matches exactly, so both the linear and quadratic weighted kappas come out to exactly 1, the same result plain Cohen's kappa would give for identical raters, since there is no partial disagreement to weight differently.

Assumption audit

Calculated from your data: the observed and expected weighted agreement under your chosen weighting scheme, and the resulting weighted kappa.
Evidence to review: which weighting scheme you selected and whether it matches how seriously you actually consider a large disagreement; quadratic weighting can produce a noticeably different kappa than linear weighting on the same data, and neither is universally correct.
You must verify: that your categories genuinely have a meaningful order (not just arbitrary numeric labels) before using this calculator; an unordered nominal category set does not have a defensible "distance" between categories for this weighting scheme to use.

What this result does not mean

Weighted kappa still measures agreement, not accuracy against an external standard, and the choice of weighting scheme changes the number without there being one objectively correct answer; report which scheme you used whenever you cite a weighted kappa value.

Limitations

This calculator requires numeric category codes that reflect a genuine order (1, 2, 3, and so on); if your data is truly nominal with no natural ordering, use the Cohen's Kappa Calculator instead, which makes no ordering assumption at all.