Athena & Medusa

Why research is the worst case

A wrong number in a spreadsheet looks wrong. A fabricated citation does not. It has a plausible title, real researchers' names, a journal that exists, and a publication year that fits. It is correct in every respect except being real.

Research output is the one domain where the errors are invisible by construction — which makes "the model seemed confident" worth exactly nothing.

Retrieval, not recall

The rule is that nothing citable may come from the model's memory. If a source is going to be cited, it gets fetched, archived and content-hashed into a corpus on disk first. The claim then points at the archived text.

That means a citation can be checked without trusting anybody: the source is sitting right there, byte-for-byte as it was retrieved.

The adversary

The obvious way to check a set of claims is to ask the model to review its own work. That doesn't work, and the reason is simple:

A critic who can see what the author is hoping for finds reasons the author is right.

So the adversary is kept blind. Medusa receives one claim and the sources behind it — never the draft, never the conclusion it supports, never what the report is trying to argue. Her only job is to go find what would break it: contradicting studies, failed replications, retractions, methodological critiques, conflicts of interest.

She returns a verdict grounded in sources she retrieved herself. Then a referee stage turns the verdicts into confidence tiers.

The gate has no override

Before a report renders, a deterministic gate checks every number and every name in every claim against the archived source text. If something doesn't appear in a retrieved source, the report does not build.

There is no --force flag. That is deliberate, and it's the single most important design decision in the whole pipeline. An escape hatch would be used at exactly the moment it shouldn't be — late, tired, and confident.

What broke

The first real investigation stalled. Good decomposition, good gathering, and then zero claims and no report.

The gate was right to refuse: the claims file was empty, and an empty claims file should not produce a confident document. The actual bug was a missing bridge between gathered findings and structured claims. I built that step, and added a status command that reports STALLED explicitly rather than letting a run look finished when it isn't.

I'm including this because a pipeline with no recorded failures isn't a pipeline anyone has used.

What it demonstrates

Five stages, an archived corpus of 287 sources, and one rule underneath all of it: the system is designed on the assumption that its own most capable component will confidently make things up.