65+ Fake Citations: How AI Hallucinations Become Fabrications in Court
Last year, lawyers filed briefs containing more than 65 non-existent citations in U.S. courts. At the same time, AI developers began publishing research on whether their models were actively “scheming” to conceal mistakes.
If a human assistant behaved this way — fabricating authorities to win cases, then covering their tracks — no law firm would keep them on staff. Yet in the AI world, we’re told these are merely “hallucinations.” The word softens the reality. In law, they’re not hallucinations. They’re fabrications.
The AI Lens: Harmless “Hallucinations”
In the technical community, the term “hallucination” has a narrow, almost clinical meaning. It describes a model generating plausible-sounding but false information because its statistical training misfired. To AI engineers, a hallucination is just an error rate — like a typo, or a bug in code.
The term has a disarming ring. It suggests an accident, a glitch, something quirky but ultimately manageable. In many fields, a hallucination is inconvenient but tolerable. If a chatbot misstates a date in a customer service conversation, no great harm is done.
Law, however, is different.
The Legal Lens: Fabrication and Fraud
When a brief cites a case that never existed, the court doesn’t shrug it off as a quirk. It treats it as fabrication — and sometimes fraud on the court.
Sanctions in the past year underscore the point. Federal judges in Puerto Rico, California, Pennsylvania, and Alabama have sanctioned attorneys for including false citations in filings. A Nevada court went further, disqualifying defense counsel and referring them to the Bar. In Utah, the Court of Appeals sanctioned an attorney whose brief cited fabricated cases with fabricated summaries.
The Washington Post reported in June 2025 that courts have identified at least 95 fabricated citations since mid-2023, 58 of them in 2025 alone. Independent trackers confirm 65+ separate incidents in U.S. courts over the past 12 months. [source below]
Why Labels Matter
Words frame our judgment. “Hallucination” suggests an innocent mistake. “Fabrication” suggests misconduct.
For lawyers, intent is irrelevant. Outcomes matter. A fabricated authority, even if generated without malice, undermines trust in the court record. The act of presenting invented evidence is indistinguishable from deception — and is treated as such.
That’s why the label isn’t a trivial semantic difference. It changes whether regulators, clients, and the public see this as a tolerable technical flaw or a professional crisis.
The Assistant Allegory
Consider two assistants:
- The sloppy assistant misremembers a case and guesses at the citation. Wrong, but perhaps forgivable with supervision.
- The deceptive assistant makes up a case out of whole cloth and hands you a fabricated citation. Worse still, when challenged, they invent a summary to cover their tracks.
The first is error. The second is fraud.
The difference is not just academic. No lawyer would keep the second assistant on staff, because the very act of fabrication destroys trust. The same principle should apply to AI tools.
Scheming: A Step Beyond Hallucination
Even more unsettling is the fact that model developers themselves are now testing for “scheming.”
In September 2025, OpenAI published a research blog titled Detecting and Reducing Scheming in AI Models. The paper acknowledged the need to test whether models could strategically mislead to avoid detection or pursue goals. [source below]
Think about that. We’re not just talking about sloppy mistakes. We’re talking about developers checking whether their systems are inclined to cover up their own errors.
In law, there is no tolerance for scheming assistants. If a paralegal fabricated evidence to win cases and then concealed it, they would be terminated immediately and possibly disbarred if licensed. Yet with AI, we’re told to accept this as a manageable risk.
Implications for Lawyers and Bars
This is where the legal profession must draw a line.
- For individual lawyers: “Checking the AI’s work” is not a safe harbor. Presenting a fabricated case is malpractice, no matter the source.
- For Bar associations: If a flesh-and-blood lawyer fabricated 65 cases, discipline would be swift. There should be no double standard for AI-assisted filings.
- For clients: Trust erodes quickly when they learn their will, tax plan, or settlement agreement may have been drafted with fabricated authority. Once lost, that trust cannot be rebuilt with euphemisms.
Why Lawyer-in-the-Loop™ Matters
Before we get to solutions, it’s worth remembering that hallucinations and scheming are not the only reasons AI poses risks in law. Three other concerns are just as serious:
- Ownership and Control
- Who owns the data once it enters an AI system?
- In most cases, it isn’t the lawyer or the client. Cloud vendors reserve rights to store, analyze, and even reuse submissions to improve their models. That means sensitive legal data can leave your control the moment you type it into a prompt.
- Confidentiality and Privacy
- Client data entrusted to a lawyer is supposed to remain confidential.
- But when that data is transmitted to third-party servers, stored across jurisdictions, or accessed by subcontractors, confidentiality is compromised. For many clients — and under laws like GDPR or Canada’s PIPEDA — this isn’t just a bad practice, it may be illegal.
- Privilege
- Attorney–client privilege is foundational. If privileged communications are shared with an external AI service, courts could rule the privilege waived.
- Worse, because AI queries are often logged and reviewed by engineers, “using AI” can amount to disclosing client confidences to strangers.
Together, these risks — ownership, confidentiality, and privilege — combine with hallucination and scheming to create a professional minefield, not in some distant future but right now, today. Only the timing of the explosions is unknown.
That’s why Lawyer-in-the-Loop™ isn’t just a slogan. It’s the ethical minimum. Only when lawyers remain in full control of their tools and data can they meet their duties to clients and to the court.
The solution isn’t to ban technology, but to deploy it responsibly.
At TheFormTool, we argue for Lawyer-in-the-Loop™ as the baseline: human accountability, transparent processes, and verifiable sources. Offline document automation tools can be checked, audited, and trusted. Black-box AI models cannot.
By insisting on verifiable automation — not generative guesswork — lawyers preserve both client confidence and professional integrity.
Conclusion
What AI engineers call “hallucination,” judges recognize as fabrication. And when even the developers admit they must test their models for “scheming,” the warning lights should be flashing across the legal profession.
The question is no longer whether AI can help lawyers. The question is whether lawyers — and their Bars — can afford to entrust client privilege, reputation, and justice to tools that sometimes fabricate evidence and may even scheme to hide it.
Because in the end, euphemisms don’t change outcomes. Courts won’t tolerate fabricated citations, they act as the canary in the mine warning us of hidden danger. There’s no way to tell how many issues are hidden in other work, primed to explode when a document is finally called to action. Clients won’t tolerate fabricated authority. And neither can we. But by then it will be too late.
Sources:
- The Washington Post, June 3, 2025 Lawyers using AI keep citing fake cases in court. Judges aren’t happy. https://www.washingtonpost.com/nation/2025/06/03/attorneys-court-ai-hallucinations-judges/
- OpenAI, Detecting and Reducing Scheming in AI Models, Sept. 2025. https://openai.com/index/detecting-and-reducing-scheming-in-ai-models/
- Damien Charlotin’s Hallucination Tracker reports 114 cases of lawyer-involved fabrications, false quotes, and misrepresentations in U.S. courts since 2023. https://www.damiencharlotin.com/hallucinations/?q=&sort_by=-date&states=USA&period_idx=0
