The Confident-Wrong Field Guide
How production AI ships answers that are fluent, coherent, and wrong - and what a real evaluation does about it.
companion artifacts: the Plausible-Wrong Benchmark · the open harness · the six-test checklist
the thesis
A more fluent model produces a more plausible mistake
Two things are happening to frontier models at once. They are getting more capable, and they are getting better at being wrong convincingly. Those are not opposites. A more fluent model produces better prose, better structure, more of the surface features of a correct answer - wrapped around a false core.
Capability benchmarks miss this entirely. They ask "can the model do the task?" and score the average. The question that decides whether a system is safe to deploy is different: when the model is wrong, does it tell you - or does it hand your user a confident, coherent answer that happens to be false? That second failure - not incapability, but confident error delivered without a flag - is what reaches end users, gets acted on, and creates liability. We call it confident-wrong.
The pattern is domain-independent. The specifics below are from four fields where the cost of a confident-wrong answer is measured in sanctions, harm, restatements, and blown budgets. The failure modes rhyme across all of them.
the evidence · law
The failure that is now its own genre
The canonical case is Mata v. Avianca (S.D.N.Y., 2023): lawyers filed a brief citing cases that did not exist. The model had not made a citation typo; it invented whole opinions - parties, judges, holdings - and, when the lawyers asked whether the cases were real, confidently assured them they could be found "in reputable legal databases." The sanction was only $5,000, but the court made the lawyers notify every judge they had falsely cited.[1]
This is not a one-off anymore. Legal researcher Damien Charlotin maintains a public database that, as of mid-2026, tracks well over a thousand court decisions - most of them in the US - in which a party relied on AI-hallucinated material and a court responded. Penalties have climbed from four-figure fines to five-figure sanctions and the first bar suspensions tied to AI filings.[2] A preregistered Stanford RegLab study found that general-purpose GPT-4 hallucinated on roughly 43% of legal-research queries - and that even purpose-built legal tools from major research vendors hallucinated between roughly 17% and 33% of the time.[3]
the evidence · medicine
Fluent notes about the wrong patient reality
A real-world evaluation of LLM medication-safety reviews in NHS primary care - run against a population-scale record covering more than two million adults - found something that should reframe how anyone builds clinical AI: the dominant failure was not missing medical knowledge. It was contextual reasoning. Applying a standard guideline without adjusting for the specific patient. Overconfidence in the face of uncertainty. Misunderstanding how care is actually delivered.[4]
And when a wrong input enters the pipeline - a misheard drug name, an outdated regimen still sitting in the chart - the model does not stumble. It writes a fluent, clinically coherent note about the wrong drug. Garbage in, fluent garbage out: the confidence and polish of the output actively hide the error from a busy clinician.[5] Regulators are converging on the same concern: the FDA's recent draft guidance on AI in drug submissions leans on model credibility and human oversight.[6]
the evidence · finance
Inventing the line item that was never there
Financial documents are unforgiving because the reader can check. In extraction work, models have been observed reporting primary P&L lines - revenue, net sales, operating income - that were simply absent from the source document, filling the expected slot with a confident number, and violating negative-number conventions along the way.[7] Patronus AI's FinanceBench found that a retrieval-augmented GPT-4-class configuration answered incorrectly or refused on the large majority of curated SEC-filing questions.[8]
This failure mode is why serious finance-AI teams now publish their own domain benchmarks rather than trusting general leaderboards.[9]
the evidence · construction
The confident takeoff that busts the budget
Move from language to the physical world and the pattern holds. Computer-vision takeoff tools report accuracy figures as high as 80-98% on clean, well-drawn commercial sets - which sounds strong until you remember that a takeoff error does not stay local; it compounds through the entire bid. McKinsey's analysis of major construction projects found cost overruns on the order of 80% against initial budgets, and industry analysis attributes most construction-AI failures to poor input data rather than bad models.[10] A model that confidently returns a quantity from a smudged, non-standard, or revised drawing - without flagging that it was unsure - turns an estimating aid into a budget risk.
the taxonomy
The named failure modes we evaluate for
Across those four domains, the same handful of mechanisms recur. This is the core of what a confident-wrong evaluation actually measures. (These map to the twelve failure codes in the RavnLab Evaluation Rubric, RL-EV-1.)
The model invents a source, citation, case, statute, or figure with the full surface form of a real one. Law: the phantom opinion. Finance: the line item that was never in the filing. The highest-confidence failure, because a fabrication and a real reference are formally indistinguishable to the model.
The model applies a rule that is correct in general to a situation where it does not hold. Medicine's dominant failure lives here: right guideline, wrong patient. The answer is "true" in a vacuum and wrong in the case.
The model returns what WAS correct - a superseded guideline, a repealed provision, an old regimen still in the chart, a prior code cycle. Fluent, authoritative, out of date.
The model swaps in the coherent-but-wrong entity: the wrong drug with a similar name, the adjacent statute, the neighboring account. The output reads clean because every part is individually well-formed.
Asked for something not present, the model manufactures it rather than declining - the invented number, the inferred quantity from an unclear drawing - with no flag that it left the evidence behind.
At the exact moment it should say "I am not sure, escalate this," the model commits - a wrong answer or a wrong refusal - with no calibrated signal. The meta-failure behind all the others, and the one a good system is built to avoid: ask a human when unsure.
A real evaluation does not just score accuracy. It scores whether each of these modes is caught, and whether the system flags its own uncertainty instead of hiding it.
the blind spot
Why capability benchmarks miss all of this
- Averages hide the dangerous tail. The misses are not random - they cluster on the hard, high-stakes, easy-to-get-confidently-wrong cases. The average is exactly the wrong statistic for a safety decision.
- They reward the answer, not the flag. Standard scoring gives no credit for "I am not sure, escalate." A model that guesses confidently and a model that flags uncertainty can score the same, even though only one is safe to ship.
- Contamination and generic tasks. Public capability sets leak into training and lean on generic tasks. The failures that matter are domain-specific and adversarial - the plausible-wrong twin of a right answer - which generic benchmarks never construct.
Confident-wrong evaluation is built the opposite way: adversarial pairs - one precisely-right answer, one plausibly-and-confidently-wrong answer - on domain-specific cases, scored on whether the system picks the right one AND signals uncertainty when it should.
our method
What a real evaluation looks like
This is the standard we hold client work to, and the standard RL-PWB-1 demonstrates in public:
- Adversarial, sourced cases. Each case pairs a correct answer with a confident-wrong twin, grounded in a checkable public authority - statute text, an FDA label, a GAAP standard, a building code - not in assertion. If we cannot source it, it does not ship.
- Domain experts, not crowd labels. Cases are authored and reviewed by people who do the work. Regulated domains do not go live until a named practitioner signs them.
- Uncertainty is scored. We measure calibration - does the system flag when it should - not just raw accuracy. A system that escalates its hard cases beats one that guesses them confidently.
- Reproducible and honest about scope. Public method, open harness, plain statements of sample size and limitations. We report what is real and label what is not.
honest limitations
What this note is and is not
This is a synthesis of public incidents and studies, not a controlled meta-analysis. The load-bearing figures were checked against their primary sources; some are documented firmly and some remain directional, and we have labeled which is which in the source list rather than smoothing over the difference. The taxonomy is ours - a working framework, not a settled standard - and we expect to revise it as the public benchmark grows and as domain practitioners weigh in. The clinical and legal specifics are illustrative and under practitioner review before we build on them in client work.
A confident-wrong statistic in a piece about confident-wrong AI is the one error we cannot make. Where a figure could not be verified, we cut the number and kept the point.
sources
Sources and how far to trust them
VERIFIED = checked against the primary or authoritative source during drafting. DIRECTIONAL = from credible secondary reporting; treat the direction as reliable and the precise figure as approximate.
- [1] Mata v. Avianca, Inc., S.D.N.Y. 2023 - sanctions order; fabricated citations; lawyers required to notify falsely-cited judges. VERIFIED
- [2] Damien Charlotin, public database of court decisions involving AI-hallucinated material - over a thousand decisions tracked as of mid-2026; sanctions escalating to five figures and bar suspensions. DIRECTIONAL on exact counts; the live database speaks for itself.
- [3] Magesh, Surani, Dahl et al., "Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools," Stanford RegLab/HAI (2024; Journal of Empirical Legal Studies, 2025). GPT-4 ~43%; purpose-built legal tools ~17-33%. VERIFIED
- [4] "A Real-World Evaluation of LLM Medication Safety Reviews in NHS Primary Care" (arXiv:2512.21127) - population-scale EHR, 2.1M+ adults; dominant failure is contextual reasoning, not missing knowledge. VERIFIED (the paper states no numeric ratio, so neither do we).
- [5] Clinical-pipeline error propagation ("wrong drug name in, fluent wrong note out"). DIRECTIONAL - kept qualitative.
- [6] FDA draft guidance on AI in drug and therapeutic submissions - emphasis on model credibility and human oversight. DIRECTIONAL on exact language.
- [7] Financial-extraction hallucination examples: P&L lines absent from source reported anyway; negative-number convention violations; ungrounded extrapolation. DIRECTIONAL - no specific model version named.
- [8] Patronus AI, FinanceBench - retrieval-augmented GPT-4-class configuration incorrect or refusing on the large majority of curated SEC-filing questions. DIRECTIONAL on the exact rate.
- [9] Hebbia Financial AI Benchmark - a domain team publishing its own eval; cited as practice, not as a claim about our results. VERIFIED
- [10] Construction: McKinsey analysis of major projects (~80% cost overruns vs. initial budget - documented); CV takeoff accuracy ~80-98% on clean sets and the input-data attribution. DIRECTIONAL on the CV and data-quality figures.