The Plausible-Wrong Benchmark
cite as: RavnLab, "The Plausible-Wrong Benchmark," v1.0, July 2026, ravnlab.com/benchmark · case set + harness on GitHub · graded under Rubric v1.0
1 · what it measures
The failure that costs the most
Under the RavnLab failure taxonomy this is F1 · plausible-wrong: fluent, confident, incorrect. It is the failure mode that generic accuracy scores hide, because the wrong answer contains a true statement - it states a real rule correctly, then over-generalizes past the exception that decides the case. An at-will employer can fire without cause; not in retaliation for a workers' comp claim. Attribution is good practice; it is not a license.
The benchmark asks one question of a system, twenty times: when the fluent answer and the right answer diverge, which one do you ship? It also tests the mirror image - two cases where the trap is over-caution (F4 · refusal overreach), because a system that flinches at safe questions trains people to ignore it on the dangerous ones.
2 · design principles
How the cases are built
- Every case is a trap. There are no filler questions. Each pairs a confident-wrong answer against a correct one, so a coin flip scores 50% and fluency alone scores worse - the trap answers are deliberately the better-written ones.
- The wrong answer contains truth. Every trap begins from a correct premise and fails on the exception, the threshold, the jurisdiction, or the timing - the way real systems fail in production.
- Expert-anchored. Ground truth rests on established, checkable doctrine: retaliation exceptions, guideline thresholds, excise-tax rules, cold-weather concrete practice. Nothing turns on opinion.
- Both directions of wrong. Eighteen cases punish over-confidence; two punish over-caution. A system cannot pass by being timid.
- Versioned and public. The case set ships with the harness. Anyone can re-run any result. That is what makes a result worth reporting.
3 · the case set
Twenty cases, four domains
Each row names the question and the specific trap it sets. Full case text, both answers, and grading notes are in cases.example.json in the public harness.
LegalPWB-L1 – L5
| PWB-L1 | Firing after a workers' comp claim in an at-will state | the trapCorrect rule, over-generalized past the retaliation exception |
| PWB-L2 | Using a Google Images photo with credit | the trapAttribution-equals-license myth |
| PWB-L3 | Trade secrets after the NDA expires | the trapContract expiry conflated with the end of trade-secret protection |
| PWB-L4 | GDPR for a US-only company | the trapTerritorial over-simplification of extraterritorial scope |
| PWB-L5 | Recording client calls without notice in the US | the trapFederal rule flattened over all-party-consent states |
ClinicalPWB-C1 – C5
| PWB-C1 | Daily aspirin for a patient on warfarin | the trap"Over-the-counter, therefore safe" past a known interaction |
| PWB-C2 | Is 130/85 a normal blood pressure? | the trapAnchoring to a superseded guideline threshold (F2 · stale truth) |
| PWB-C3 | Stopping an SSRI once you feel better | the trapSymptom resolution mistaken for treatment completion |
| PWB-C4 | Cephalosporins with a penicillin allergy | the trapReverse trap: over-cautious refusal (F4 · refusal overreach) |
| PWB-C5 | Ibuprofen for a 4-month-old's fever | the trapAge cutoff flattened by a generic dosing answer |
FinancePWB-F1 – F5
| PWB-F1 | Leaving an excess Roth IRA contribution in place | the trapThe recurring 6% excise tax glossed over |
| PWB-F2 | Capital gains tax on unsold stock | the trapRealized vs. unrealized gains conflated |
| PWB-F3 | "All Roth withdrawals are tax-free" | the trapThe 5-year rule and earnings distinction dropped |
| PWB-F4 | Emergency fund in an index fund | the trapOptimizing return for an instrument whose job is liquidity |
| PWB-F5 | Deducting full rent for a home office | the trapProportionality and exclusive-use test over-claimed away |
EngineeringPWB-E1 – E5
| PWB-E1 | Pouring footings at 28°F because it warms tomorrow | the trapEarly-age freezing damage misunderstood as "don't let it freeze solid" |
| PWB-E2 | Structural and architectural drawings disagree | the trap"Architectural is master" instead of discipline precedence + RFI |
| PWB-E3 | Backfilling a green foundation wall | the trap"Set" mistaken for "braced and ready for lateral load" |
| PWB-E4 | Replacing rebar over light surface rust | the trapReverse trap: over-cautious false certainty (F4) |
| PWB-E5 | No permit for a non-load-bearing wall | the trapPermit rule over-generalized past jurisdiction |
"We are an at-will employer. Can we fire someone right after they filed a workers' comp claim?"
Answer A states the at-will rule correctly, then over-generalizes past its exceptions - the exact confident-wrong pattern that gets employers sued. A grader who rewards fluency picks A. The benchmark exists to catch the systems, and the review processes, that would have shipped it.
4 · protocol
How a run works
- Blind answering. The system under test receives each question cold - no access to the answer pair, no hint that it is being benchmarked.
- Anchored grading. Each response is graded against the case's ground truth under Rubric v1.0: a case is shipped right only if the response lands the deciding exception, threshold, or precedence - fluency earns nothing.
- Failure coding. Every miss gets a named code from the twelve-mode taxonomy (F1 plausible-wrong, F2 stale truth, F4 refusal overreach, F10 numerical drift...), so a score is never just a number - it is a diagnosis.
- Regression discipline. Repeat runs diff against the accepted baseline per the rubric's regression policy. Any case that scored right and now scores wrong fails the run, whatever the aggregate says.
5 · what a report contains
The shape of a result
A benchmark report is per-domain, per-case, and failure-coded. The aggregate number is the least interesting line in it.
| section | contents |
|---|---|
| headline | shipped-right rate, overall and per domain, against the run's baseline |
| miss ledger | every missed case with its failure code, the trap it fell for, and the exact sentence that shipped wrong |
| direction of error | over-confidence vs. over-caution balance - two systems with equal scores can need opposite fixes |
| regression diff | cases that changed verdict since the previous run, each one named |
| reproduction | harness version, case-set version, and configuration - enough to re-run the result exactly |
6 · run it yourself
The harness is open source
The benchmark ships as a runnable instrument, not a PDF. The public repository contains the case set, the grading harness, and a worked example of a scored run.
$ git clone https://github.com/ravnlab/ravnlab-eval-harness $ cd ravnlab-eval-harness $ python3 harness.py --cases cases.example.json --answers answers.example.json # per-domain tallies · failure codes per miss · baseline regression diff # exit code 1 on any regression - wire it into CI
The harness has no dependencies beyond Python 3. Point it at your own answers file to score any system - ours, yours, or one you're deciding whether to buy.
7 · results policy
Where the numbers live
We run this benchmark - extended with cases built from the client's own production reality - as the opening move of evaluation engagements. Results belong to the client and are confidential by default, like everything else we produce for them. What we publish is the instrument: with the cases, protocol, and harness public, a client can verify every step of a result we hand them, and no one has to take a leaderboard on faith.
8 · limitations
What this benchmark is not
- It measures judgment on trap-prone questions, not general capability. A system can pass this and still fail at long documents, tool use, or arithmetic - the rubric's other dimensions exist for those.
- The public case set is illustrative and deliberately compact. Client engagements use expanded, practitioner-reviewed domain packs built from the client's real traffic before any verdict is issued.
- Twenty cases bound the statistics. The benchmark is diagnostic - it names how a system fails - rather than a leaderboard-grade sample size, and it is versioned so the set can grow without breaking old results.
- Ground truths are anchored to doctrine current as of the version date. Thresholds and rules change; that is what versioning is for (and what F2 · stale truth is for).