StatReason

Intraclass Correlation Calculator

Quantify how reliably a set of raters or repeated measurements agree, using an explicit ICC model rather than a single generic "reliability" number, since different study designs genuinely need different ICC formulas.

What this answers

This calculator answers "how much of the variation in these ratings comes from real differences between subjects, versus disagreement between raters or measurement error?" The answer depends heavily on your actual design, which is why an explicit model choice is required rather than one universal formula.

How it is calculated

Every model starts from the same subject-by-rater variance decomposition: how much variability sits between subjects, how much sits between raters, and how much is leftover error. ICC(1,1) assumes each subject was rated by a different, effectively random set of raters, folding rater differences into the error term. ICC(2,1) treats the same fixed set of raters as a random sample from a larger population of possible raters and targets absolute agreement, penalizing systematic rater bias. ICC(3,1) treats the raters as the only raters of interest (a fixed effect) and targets consistency, which does not penalize a rater who is consistently higher or lower than the others by a fixed amount. This calculator implements the standard Shrout and Fleiss (1979) formulas for all three models, plus their average-measures counterparts.

Worked example

For raters who all agree perfectly on the relative ranking and absolute level of every subject (identical ratings across every rater for a given subject), every model, ICC(1,1), ICC(2,1), and ICC(3,1) alike, correctly returns exactly 1, since there is no disagreement of any kind for any of the three models to detect.

Assumption audit

Calculated from your data: the between-subjects, between-raters, and error mean squares feeding into your selected ICC model, and the resulting ICC value.
Evidence to review: whether raters show a consistent bias relative to each other (some always rating higher than others); ICC(2,1) penalizes this while ICC(3,1) does not, so the two models can give meaningfully different answers on the same data.
You must verify: which model actually matches your design before reading the result; choosing a model based on which number looks best, rather than which model fits your study, defeats the purpose of an explicit model choice.

What this result does not mean

A high ICC means the ratings are reliable under your chosen model's specific definition of reliability, not that the ratings are valid or accurate relative to a true underlying value; reliability and validity are different properties, and this calculator addresses only the former.

Limitations

This calculator requires every subject to have a rating from every rater (a complete, balanced matrix with no missing cells) and computes single-measure and average-measure variants only for k raters per subject, k, 1, or 2 fixed reliability targets, not a general formula for an arbitrary subset of raters.