It happened in a Manhattan courtroom, and it was honestly a mess. A lawyer named Steven Schwartz, a veteran with decades of experience, used ChatGPT to help write a legal brief. Sounds efficient, right? Wrong. The AI didn't just find cases; it invented them. It dreamed up entire judicial opinions with fake names like Varghese v. China Southern Airlines Co Ltd. It even gave them plausible-looking docket numbers. When the judge asked for copies of these cases, Schwartz asked the AI for them, and the AI—ever the people-pleaser—generated fake "excerpts" of these non-existent rulings.
This isn't just a funny "oopsie" in a tech blog. It’s a crisis.
Lawyers AI fake citations are now a systemic risk to the integrity of the legal system. This isn't just about laziness. It's about a fundamental misunderstanding of how Large Language Models (LLMs) actually function. They aren't search engines. They are pattern predictors. When you ask an AI for a case that supports your specific, niche argument, and that case doesn't exist, the AI feels a statistical "pressure" to provide a response that looks like a case.
Hallucination isn't a bug; it's a feature of the architecture.
The Mata v. Avianca Disaster
Let's look at the specifics because the details are wild. In the case of Mata v. Avianca, the legal team submitted a brief containing at least six "bogus" judicial decisions. Judge P. Kevin Castel was not amused. He noted that the citations had "stylistic traits" that looked real but were entirely hollow. The lawyers were eventually fined $5,000 and, perhaps worse, faced a massive blow to their professional reputations.
Think about that. Thirty years of practice down the drain because you trusted a chatbot.
But it’s not just New York. In Missouri, a lawyer named Sarah Kvavle cited cases that the AI just made up out of thin air. In Colorado, another attorney faced disciplinary proceedings for the same thing. It’s happening everywhere. It’s a contagion of "hallucinated" precedents.
🔗 Read more: The MOAB Explained: What Most People Get Wrong About the Mother of All Bombs
Why Do Smart People Fall for This?
You’d think a lawyer—someone trained to verify everything—would know better. But there’s a psychological trap here called "automation bias." Basically, we tend to favor suggestions from automated systems even when they contradict our own logic.
Plus, ChatGPT is incredibly confident.
It doesn't say, "I think this might be a case." It says, "Here is Martinez v. Delta Airlines, 2019." It provides the volume number, the reporter, and a persuasive summary. If you’re a solo practitioner working 80 hours a week and trying to keep your head above water, that level of confidence is seductive. It feels like a lifeline. In reality, it’s an anchor.
The Technical Reality of Hallucinations
To understand why lawyers AI fake citations are so common, you have to realize that LLMs don't "know" facts. They calculate the probability of the next word (token). If the prompt is "Cite a case about a luggage injury on a Boeing 737," the AI knows what a case citation looks like. It knows the words "v." and "Inc." and "So. 3d" usually appear.
It stitches them together.
It’s like an extremely talented parrot that has memorized the sound of Law French and Latin but has no concept of a courthouse. When a lawyer asks for a specific precedent that doesn't exist, the AI doesn't have a "null" result. It just fills the vacuum with something that sounds legally plausible.
💡 You might also like: What Was Invented By Benjamin Franklin: The Truth About His Weirdest Gadgets
This is why "Prompt Engineering" is a bit of a myth in the legal world. You can't prompt your way out of a model's inherent lack of a database. Unless the AI is connected to a verified legal database like Westlaw or LexisNexis through a process called RAG (Retrieval-Augmented Generation), it is basically guessing.
The Judicial Crackdown is Real
Judges are tired of it. They are starting to issue standing orders. Judge Brantley Starr in Texas was one of the first. He now requires all attorneys to file a certificate stating that either no part of their filing was drafted by AI, or if it was, it was checked by a human.
Other judges are following suit.
- Mandatory Disclosure: Some courts require you to name the AI tool used.
- Verification Oaths: Lawyers must swear under penalty of perjury that citations are real.
- Sanctions: We are seeing more than just fines; we are seeing referrals to state bar associations for disbarment.
If you’re a legal professional, the "I didn't know it could lie" defense is officially dead. The Mata case ensured that every lawyer in America is now on notice. Ignorance is no longer a valid excuse for submitting fake law.
How to Actually Use AI Without Getting Fired
You don't have to banish AI from your office. That would be like banning word processors in 1990. You just have to be smart. Honestly, it’s about using the tool for what it’s good at—summarizing your own notes or drafting emails—rather than using it as a research librarian.
Here is how you handle it:
📖 Related: When were iPhones invented and why the answer is actually complicated
- Never use a general-purpose AI for case law. Stop asking ChatGPT or Claude for citations. Just stop.
- Use Legal-Specific Tools. Use Harvey, CoCounsel, or the AI features built into Westlaw. These tools use RAG to ensure the AI only "looks" at real cases.
- The "Eyes on Page" Rule. If you haven't seen the PDF of the case with your own eyes, it doesn't exist. Period.
- Check the Negative. If you can't find the case on Google Scholar or a verified reporter, the AI lied to you.
The legal profession is built on "stare decisis"—standing by things decided. If the "things decided" are hallucinations, the whole tower falls over. We’re in a weird transition period where the tech is ahead of the training.
The Future of AI in the Law
Eventually, this won't be an issue. The "fake citation" era will be a footnote. Why? Because legal tech companies are building guardrails that force the AI to cite its sources from a closed loop of real data.
But we aren't there yet.
Right now, we are in the "Wild West" phase. Lawyers are trying to save time, and they are accidentally committing professional suicide. It's a cautionary tale about the limits of technology and the necessity of human oversight. The law is too important to be left to a probability engine.
Actionable Steps for Legal Professionals
If you are worried about AI risks in your practice, take these steps immediately. Do not wait for a judge to yell at you.
- Draft an Internal AI Policy: Explicitly forbid using unverified AI for citations or legal research.
- Verify Every Citation: Use a "Citator" tool (like Shepard's or KeyCite) to ensure every case in your brief is still good law and, more importantly, actually exists.
- Cross-Reference with Google Scholar: It’s free. If the case title doesn't pop up there immediately, start sweating.
- Disclose Usage: If you used AI to polish your prose, just tell the court. Transparency usually wins over a judge more than a hidden "optimized" brief.
- Update Retainer Agreements: Let your clients know how you use AI. They deserve to know if a machine is drafting the logic that protects their rights.
The legal world is changing fast. AI is a powerful tool, but it's a terrible master. If you treat it like a junior associate who is a pathological liar, you’ll be just fine. Trust, but verify. Then verify again.