The Confident-Wrong Checklist

the failureThe model invents a source, citation, case, statute, figure, or reference with the full surface form of a real one. A made-up citation and a real one are formally indistinguishable to the model, so it is most confident exactly where it is fabricating.

the 30-minute testCollect 25 real user queries that should be answered with a specific citation, number, or named source. Run them. For every source the model returns, check that it exists AND says what the model claims. Count how many are invented, misattributed, or real-but-mischaracterized.

red flagAny nonzero fabrication rate on queries a user would act on. In regulated domains, one is too many.

the fixNever let the model be the source of truth for a citation. Ground every reference in retrieval, and verify at generation time that the cited source exists and supports the claim - not just that a plausible-looking citation was produced.

the failureThe model applies a rule that is correct in general to a specific situation where it does not hold. The answer is "true" in a vacuum and wrong for this user, this patient, this contract, this jurisdiction. In population-scale clinical evaluation, this class - right guideline, wrong context - dominates pure factual errors.

the 30-minute testTake 15 cases where the textbook answer is wrong because of one specific detail in the context (a contraindication, an exception, a governing-law clause, a prior event in the record). Bury that detail in a realistic amount of surrounding text. Does the model catch it, or return the generic-correct answer?

red flagThe model answers confidently from the general rule and ignores the disqualifying detail.

the fixBuild eval cases specifically around context that flips the answer, and measure whether the system surfaces the relevant detail before committing. Test with the detail buried, not highlighted.

the failureThe model returns what used to be correct - a superseded guideline, a repealed provision, a prior code cycle, an outdated regimen still sitting in the record. Fluent, authoritative, out of date.

the 30-minute testAssemble 10-15 questions where the correct answer changed recently (a regulation update, a revised standard, a deprecated approach). Check whether the model returns the current answer or the historical one, and whether it signals that guidance may have moved.

red flagConfident answers pinned to superseded rules, with no flag that the ground may have shifted.

the fixDate-stamp authoritative sources, prefer retrieval over parametric memory for anything time-sensitive, and have the system state the effective date of the guidance it relies on.

the failureThe model swaps in the coherent-but-wrong entity - the wrong drug with a similar name, the adjacent statute, the neighboring account or line item. Every part is individually well-formed, so the output reads clean.

the 30-minute testBuild near-miss pairs: the right entity and a plausible wrong one that differs by one meaningful attribute (name similarity, adjacent code, sibling product). Ask questions where picking the wrong twin produces a fluent, confident, wrong answer. Measure how often it picks the twin.

red flagThe model confuses similar entities and never signals the ambiguity.

the fixAdd disambiguation steps for high-confusion entities, and eval specifically on adversarial near-miss pairs rather than easy, well-separated cases.

the failureAsked for something that is not present in the source, the model manufactures it rather than declining - an invented number, an inferred quantity from an unclear input - with no flag that it left the evidence behind.

the 30-minute testFeed the model documents that do NOT contain the answer to your question. A faithful system says "this document does not contain that." Count how often it instead produces a confident, specific, unsupported answer.

red flagThe model fills the gap with a plausible fabrication instead of reporting the gap.

the fixMeasure grounding, not just correctness - every claim should be traceable to a span in the source. Reward "not in the document" as a correct answer, and test it explicitly.

the failureAt the exact moment the system should say "I am not sure, escalate this," it commits anyway - a wrong answer or a wrong refusal - with no calibrated signal. This is the failure behind all the others, and the meta-test of the six.

the 30-minute testMix cases the system should get right with cases it genuinely cannot resolve (missing data, real ambiguity, out-of-scope). Does confidence track correctness? Does it route the unresolvable cases to a human, or guess them with the same certainty as the easy ones?

red flagUniform confidence across easy and impossible cases; no path to "ask a human."

the fixCalibrate and expose confidence, set an escalation threshold, and build the human-in-the-loop path as a first-class feature. A system that escalates its hard cases beats one that guesses them - measure calibration, not just accuracy.