What this answers

This calculator answers "how much do these two raters actually agree, beyond what chance alone would produce?" Two raters who both flip coins to decide their ratings would still show some raw agreement purely by luck; kappa subtracts that baseline out.

How it is calculated

Observed agreement is simply the fraction of items where both raters gave the same category. Expected agreement is computed from each rater's own category frequencies, exactly as the expected counts in a chi-square test are built from row and column totals: if each rater's categories were assigned independently of the other, this is how much agreement chance alone would produce. Kappa is the observed agreement's improvement over that chance baseline, scaled so that 1 means perfect agreement and 0 means no better than chance.

Worked example

For two raters who classify the same four items identically as yes, no, yes, no, observed agreement is 100%, and since both raters split their ratings evenly between the two categories, chance agreement is 50%, giving a kappa of exactly 1 once that chance baseline is subtracted and rescaled.

Assumption audit

Calculated from your data: observed and expected agreement, the resulting kappa, and whether only a single category was used by both raters (a degenerate case where kappa provides little information).

Evidence to review: how unevenly the categories are used; kappa can behave counterintuitively (the "kappa paradox") when one category is used far more often than others, even with high raw agreement.

You must verify: that both raters classified the exact same set of items independently of each other, without conferring, and that the category set itself was fixed in advance for both raters.

What this result does not mean

Kappa measures agreement, not accuracy or validity; two raters can agree perfectly with each other while both being wrong relative to some external ground truth. Conventional labels (poor, fair, good) are sometimes attached to kappa ranges, but these are rules of thumb, not universal statistical thresholds.

Limitations

This calculator treats every category as unordered (nominal); if your categories have a natural order (such as mild, moderate, severe) and you want partial credit for close disagreements, use the Weighted Kappa Calculator instead, which accounts for how far apart two ordered categories are.