Law Firms Caught and Punished for Passing Around “Bogus” AI Slop in Court

“No attorney or staff member at either firm apparently cite-checked or otherwise reviewed that research before filing the brief.”

/ Artificial Intelligence/ Ai Chatbots/ Ai Slop/ Artificial Intelligence

Image by Getty / Futurism

As The Verge reported this week, the court filing in question was a brief for a civil lawsuit against the insurance giant State Farm. After its submission, a review of the brief found that it contained “bogus AI-generated research” that led to the inclusion of “numerous false, inaccurate, and misleading legal citations and quotations,” as judge Michael Wilner wrote in a scathing ruling.

According to the ruling, it was only after the judge requested more information about the error-riddled brief that lawyers at the firms involved fessed up to using generative AI. And if he hadn’t caught onto it, Milner cautioned, the AI slop could have made its way into an official judicial order.

“I read their brief, was persuaded (or at least intrigued) by the authorities that they cited, and looked up the decisions to learn more about them — only to find that they didn’t exist,” Milner wrote in his ruling. “That’s scary.”

“It almost led to the scarier outcome (from my perspective),” he added, “of including those bogus materials in a judicial order.”

A lawyer at one of the firms involved with the ten-page brief, the Ellis George group, used Google’s Gemini and a few other law-specific AI tools to draft an initial outline. That outline included many errors, but was passed along to the next law firm, K&L Gates, without any corrections. Incredibly, the second firm also failed to notice and correct the fabrications.

“No attorney or staff member at either firm apparently cite-checked or otherwise reviewed that research before filing the brief,” Milner wrote in the ruling.

After the brief was submitted, a judicial review found that a staggering nine out of 27 legal citations included in the filing “were incorrect in some way,” and “at least two of the authorities cited do not exist.” Milner also found that quotes “attributed to the cited judicial opinions were phony and did not accurately represent those materials.”

As for his decision to levy the hefty fines, Milner said the egregiousness of the failures, coupled with how compelling the AI’s made-up responses were, necessitated “strong deterrence.”

“Strong deterrence is needed,” wrote Milner, “to make sure that lawyers don’t respond to this easy shortcut.”

More on lawyers and AI: Large Law Firm Sends Panicked Email as It Realizes Its Attorneys Have Been Using AI to Prepare Court Documents