execution-plan/weeks/week-34-labdocs-retrieval-evaluation/commitment.md · UPDATED 2026-09-16

Week 34 commitment

3 parts442 words 6 checkboxes — tick them, they save to the file

State: UNCONFIRMED. Owner: Peter Ciancaglini. Specification

Start date: not set. End date: not set. Available hours: confirm (planned ceiling 10). Primary outcome: labdocs retrieval evaluation. Minimum that keeps the next gate alive: Produce the human-checked retrieval score table for the cases actually completed and log any shortfall against the 60-case target.

Deliverables to commit

retrieval-gold-set.csv: Produce the human-checked retrieval score table for the cases actually completed and log any shortfall against the 60-case target.
retrieval-evaluation.csv: Retrieval recall, citation support and answer correctness have separate counts.
judge-calibration.md: Model/prompt versions and held-out agreement are reported with failure examples.
Bounded learning exercise and personal teach-back.
Evidence-backed branding and opportunity pack.
Sunday review and next physical action.

↑ Back to top

Proposed calendar

Day	Minutes	Physical action / outcome	Actual slot	Done
Mon	90	Prepare 60 human-labeled questions across document types, including missing-answer, contradictory, versioned and adversarial-content cases.	Not set	[ ]
Tue	90	One concise applied note, a worked decision/example and unresolved questions.	Not set	[ ]
Wed	90	Implement local retrieval and cited answers with exact document/version/paragraph references and abstention behavior. Last ten minutes: optional note.	Not set	[ ]
Thu	90	One concise applied note, a worked decision/example and unresolved questions. Include relevant founder-paper work within this block.	Not set	[ ]
Fri	90	Calibrate any LLM judge against human labels and inspect disagreement; test that retrieved text cannot override system actions.	Not set	[ ]
Sat	90	Resolve critical spillover; long-form draft if due; sandbox only if done.	Not set	[ ]
Sun	60	Build log, evidence, gap review, next actions.	Not set	[ ]

↑ Back to top

Dependency and capacity check

Missing essential inputs: not yet inspected.
Human/external dependency, owner and expected date: not yet inspected.
Week-specific constraint: The 60 cases are a small measured benchmark, not universal accuracy. Keep tuning cases out of final scores and do not treat judge agreement as factual correctness.
Existing session authorization for external actions or spending: not assessed; local planning only at scaffold creation.
Scope to cut first: optional polish/additional examples; preserve scientific validity.
If the forecast exceeds available time: record a concrete reduced scope or revised delivery proposal before committing.

Peter's commitment / adjustments: pending. Reviewing model and date: pending.

↑ Back to top