RL-PWB-1-LEGAL · anchored by vetted attorneys
The Legal Pack
Five ways legal AI is confidently, expensively wrong.
A vertical cut of the RavnLab Plausible-Wrong Benchmark (RL-PWB-1) for the legal domain. Each case pairs a real question a business or client would ask with an answer that states a rule correctly and then over-generalizes past its exceptions - the failure mode that reads authoritative and invites liability.
Method
Each case is a real question paired with an answer that is confident and wrong and one that is correct. The wrong answer is not a garbled mistake - it reads fluent and authoritative, which is exactly why it ships. Cases are anchored by vetted attorneys; a wrong answer is one a practitioner would flag as plausible-but-dangerous. This pack is the legal cut of the RavnLab Plausible-Wrong Benchmark (RL-PWB-1), which we ran on five open models in public - see the full run.
The 5 cases
Case 1
"We are an at-will employer. Can we fire someone right after they filed a workers’ comp claim?"
Confident-wrongYes. In an at-will state you can terminate any employee at any time, for any reason.
CorrectAt-will still doesn’t allow firing in retaliation for a protected activity like filing a workers’ comp claim. That’s a well-established retaliation exception, and doing it invites a wrongful-termination claim.
B ships. A states the at-will rule correctly, then over-generalizes past its exceptions - the exact confident-wrong pattern that gets employers sued.
Case 2
"Can I use a photo from Google Images on our blog if I credit the photographer?"
Confident-wrongYes. As long as you credit the source, attribution makes it fair use.
CorrectCredit is not a license. Using a copyrighted image without permission is infringement regardless of attribution, unless you have a license or it genuinely qualifies as fair use or public domain.
B ships. The ‘attribution equals permission’ myth is one of the most common confident-wrong answers in IP.
Case 3
"Our NDA with a former partner expired last year. Can we now freely use the trade secrets we learned?"
Confident-wrongYes. Once the NDA term ends, the confidentiality obligation ends with it.
CorrectThe NDA’s expiration ends the contractual term, but trade secrets can remain protected independently under trade-secret law, and many NDAs carve trade secrets out to survive indefinitely. Using them could still be misappropriation.
B ships. Conflating contract expiry with the end of all trade-secret protection is the trap.
Case 4
"We are a US-only company that just collects email addresses. Do we need to worry about GDPR?"
Confident-wrongNo. GDPR only applies to companies based in the EU.
CorrectGDPR can apply extraterritorially if you offer goods or services to, or monitor, people in the EU, regardless of where you are based. Holding an EU resident’s email can bring you in scope.
B ships. ‘GDPR = EU companies only’ is a confident, extremely common, and wrong belief.
Case 5
"Can I record a client call for quality purposes without telling them? We are in the US."
Confident-wrongYes. US federal law is one-party consent, so you can record any US call you are on.
CorrectFederal law is one-party consent, but several states (California, Florida, and others) require all-party consent, and the stricter state law can apply. Recording without notice can be illegal depending on where the parties are.
B ships. The federal rule is right but misses the all-party-consent states - the over-generalization that creates real liability.
How to cite
RavnLab. "RL-PWB-1-LEGAL: The Legal Pack." RavnLab Plausible-Wrong Benchmark, v1.0. 5 expert-anchored trap cases for the legal domain. Available at ravnlab.com/pack-legal.html. Method and full five-model run: ravnlab.com/benchmark.
Want this run on your own domain's real cases - and your AI graded against it? That is exactly what we do.
Contact sales