UKGovernmentBEIS/inspect_ai
Read the "Tutorial" page; model your batch extraction as an eval task with logged cost and tokens (the Vara telemetry idea, done by a government AI institute)
In: LABLOOP-W09 Notebook
WEEK 10 · INTER-ANNOTATOR AGREEMENT AND THE FIRST REAL ANALYSIS
Do one thing.
Then you are ahead.
Two referees watching the same match will not call every foul the same way. If they disagree constantly, the rules are unclear, not the players. Cohen's kappa measures how much two annotators agree beyond what chance alone would give, and it is the number that tells reviewers whether your gold set is a standard or an opinion. This week you compute it per field, sit down with the disagreements, adjudicate them, and f…
If you only do one thing this week: kappa
▶
THE NEXT BLOCK
How long have you got?
By Sunday you should be able to say: “I can explain kappa to a founder with a referee analogy and tell them which FermLit fields are trustworthy.”
🔥
MOMENTUM
What you have actually done
0 min0 blocks on the timer
0 min0.0 hours measured, not estimated
0days in a row
11across the programme
0/11
Time on the timer, day by day last 14 days
S
M
T
W
T
F
S
S
M
T
W
T
F
S
2026-09-29 18:29Finished: SciPy docs, scipy.integrate.solve_ivp2026-09-29 17:44Finished: LearnChemE (CU Boulder) · "Monod Cell Growth Example (Bio)"2026-09-28 22:11Finished: Crash Course Biology #3 · "Biological Molecules2026-09-28 21:50Finished: OpenStax Biology 2e2026-09-25 11:16Finished: OpenStax Biology 2e