WEEK 10 · INTER-ANNOTATOR AGREEMENT AND THE FIRST REAL ANALYSIS

Do one thing.
Then you are ahead.

Two referees watching the same match will not call every foul the same way. If they disagree constantly, the rules are unclear, not the players. Cohen's kappa measures how much two annotators agree beyond what chance alone would give, and it is the number that tells reviewers whether your gold set is a standard or an opinion. This week you compute it per field, sit down with the disagreements, adjudicate them, and f…