Cohen's kappa

Also written: kappa

A measure of how much two labelers agree beyond what chance would produce: 0 means no better than chance and 1 means perfect. It stops a judge that always says "correct" from looking good when most answers are correct.