Two New York Lawyers and Their Firm Paid $5,000 in 2023 for Six Court Cases ChatGPT Invented, and by October 2026 a Public Tracker Listed 2,202 Court Decisions Involving AI Hallucinations

A language model hallucinates when it writes fluent, confident text that is false. On 22 June 2023 Judge Castel of the Southern District of New York ordered two lawyers and their firm to pay $5,000 because a brief they filed cited six court decisions that ChatGPT had invented. A database kept by Damien Charlotin, a French lawyer and data scientist in the Fifth Circuit’s description, listed 2,202 court decisions involving hallucinated material as of 19 October 2026. Of those, 1,501 were from the United States and 16 from India.

What researchers mean by the word

The term comes from a 2022 survey by Ziwei Ji and colleagues at the Hong Kong University of Science and Technology, later printed in ACM Computing Surveys. They define hallucination in text generation as output that is nonsensical or unfaithful to the source content. The word comes from psychology, where it means an unreal perception that feels real, and the authors note that hallucinated text reads as fluent and natural. An intrinsic hallucination contradicts the source. An extrinsic one adds something the source neither supports nor contradicts.

OpenAI’s GPT-4 report draws a similar line. A closed-domain hallucination is made-up detail in a task where the model was told to use only the text supplied. An open-domain hallucination is false information about the world, offered with confidence. A brief that cites a case nobody decided is the second kind.

Why a model writes false text that reads well

GPT-4’s technical report describes the base model as pre-trained to predict the next token, a short chunk of text, in a document. Nothing in that task checks a claim against the world. How a large language model is trained walks through the stages.

A paper dated 4 September 2025 by Adam Tauman Kalai and three co-authors, at OpenAI and Georgia Tech, argues that this is enough to produce errors even from error-free training text. They asked an open model for Kalai’s birthday, told it to answer only if it knew, and got three different wrong dates in three attempts. For facts with no learnable pattern, such as birthdays, the hallucination rate after pretraining should be at least the share of facts that appear only once in the training data. If 20 percent of birthday facts appear exactly once, they write, a base model should get at least 20 percent of birthday questions wrong.

Their second argument concerns grading. Most benchmarks score strict accuracy, and an answer of “I don’t know” earns nothing, so guessing never scores worse than abstaining. Models, the paper says, are optimised to be good test-takers. India’s JEE, NEET and GATE exams, it notes, use or have used penalties for wrong answers, and it proposes writing a confidence target into benchmark instructions. At a target of 0.75, a mistake costs 2 points and a correct answer earns 1. The benchmarks problem covers other flaws in how models are scored.

Mata v. Avianca: six invented cases and a chatbot that said they were real

Roberto Mata sued the airline Avianca over a flight injury. On 1 March 2023 his lawyer Peter LoDuca filed a brief opposing Avianca’s motion to dismiss the claim as time-barred. LoDuca signed it, but his colleague Steven Schwartz had written it, and it quoted decisions that Judge Castel’s order later found did not exist. Avianca’s reply of 15 March said its lawyers could not find most of them.

The order records that Schwartz’s firm used a database, Fastcase, with limited access to federal cases, and that he had never used ChatGPT. In his words in the record, he “falsely assumed” it “was like a super search engine called ChatGPT”. The fake decisions were Varghese, Miller, Petersen, Shaboon, Martinez and Durden. The clerk of the Eleventh Circuit confirmed that Varghese v. China Southern Airlines was not a ruling of that court. Schwartz asked ChatGPT whether Varghese was real, and according to his own declaration it answered that the case “does indeed exist” and could be found on Westlaw and LexisNexis.

Judge Castel wrote that “there is nothing inherently improper about using a reliable artificial intelligence tool for assistance”. He found bad faith in the lawyers’ conduct after the problem surfaced: they did not come clean until 25 May, after a show-cause order. The court imposed a $5,000 penalty jointly on LoDuca, Schwartz and the firm, and ordered letters to Mata and to each judge whose name had been attached to a fake opinion. It declined to compel an apology.

Park v. Kim and Johnson v. Dunn: a referral, then a disqualification

The Second Circuit decided Park v. Kim on 30 January 2024. It affirmed the dismissal of a civil suit and separately addressed the plaintiff’s lawyer, Jae S. Lee, who had cited a non-existent state court decision in her reply brief. Lee said she asked ChatGPT for a supporting case after failing to find one, and did not read it. The panel wrote that Rule 11, the federal rule under which a signature certifies that a filing’s legal claims are warranted by existing law, requires lawyers to “read, and thereby confirm the existence and validity of, the legal authorities on which they rely”. It referred Lee to its Grievance Panel and ordered her to give her client a copy of the decision. It imposed no fine.

In Johnson v. Dunn, an incarcerated plaintiff, Frankie Johnson, told the Northern District of Alabama that lawyers for the state’s former corrections commissioner, Jefferson Dunn, had cited cases that were not real. Judge Anna Manasco found five problematic citations across two motions. Three attorneys from the Butler Snow law firm confirmed in writing and at a hearing that ChatGPT had produced them. One, Matthew Reeves, had asked it for citations without checking, and said he knew his firm’s policy limited the use of AI.

Her order of 23 July 2025 says the citations were “completely made up” and treats the usual reprimand and modest fine as too weak a deterrent. She publicly reprimanded Reeves, William Cranford and William Lunsford, disqualified all three from the case, ordered the opinion published in the Federal Supplement, a printed series of federal district court rulings, and referred them to the Alabama State Bar. She did not suspend them from practice, and released two other lawyers and the firm without sanction.

India: the Supreme Court says a decision on fake judgments is misconduct

On 27 February 2026 the Supreme Court of India, in Gummadi Usha Rani v. Sure Mallikarjuna Rao, took up a trial court order that relied on four decisions, including Subramani v. M. Natarajan, cited as (2013) 14 SCC 95. The petitioners said the decisions were fake, and in the Supreme Court’s account the Andhra Pradesh High Court realised they were AI generated, added a word of caution and still decided the case on its merits. Justices Pamidighantam Sri Narasimha and Alok Aradhe barred the trial court from proceeding on the Advocate Commissioner’s report and issued notice to the Attorney General, the Solicitor General and the Bar Council of India. Their order says a decision resting on such judgments “is not an error in the decision making. It would be a misconduct”. Later 2026 rulings are covered in AI in Indian courts and government.

The Stanford study of tools sold as hallucination-free

Legal research vendors had said that retrieval-augmented generation, in which the system first fetches relevant documents and then writes its answer from them, avoids the problem. Magesh and five co-authors at Stanford and Yale quote vendor claims of “eliminating” and “avoid[ing]” hallucinations and of “hallucination-free” citations. They ran over 200 preregistered legal queries through Lexis+ AI, Westlaw’s AI-Assisted Research, Thomson Reuters’ Ask Practical Law AI and GPT-4, counting a response as a hallucination if it contained incorrect information or falsely said a source supports a statement.

Each of the three vendor tools hallucinated between 17 percent and 33 percent of the time. Lexis+ AI and Ask Practical Law AI did so on more than one query in six, and Westlaw on about one in three. GPT-4 hallucinated on 43 percent, according to the paper’s Figure 4. The authors conclude that retrieval reduces hallucination compared with GPT-4 but leaves it substantial.

What has cut the error rate, with the numbers

Retrieval has the oldest record. A 2021 paper from Facebook AI Research reports that its best retrieval-augmented chatbots reduced hallucinated responses on two conversation tasks by over 60 percent. Kalai’s team cautions that search is no cure, because grading still rewards a guess whenever the search finds no confident answer.

Training changes help too. OpenAI’s InstructGPT paper reports that on closed-domain tasks the tuned models made up information about half as often as GPT-3, a 21 percent rate against 41 percent. The GPT-4 report says its model scored 19 percentage points higher than the latest GPT-3.5 on an internal adversarial factuality test, and its system card gives 19 points for open-domain and 29 points for closed-domain hallucination.

Checking a draft also helps. In Meta’s Chain-of-Verification method the model drafts an answer, writes fact-checking questions, answers them separately and then revises. On list questions drawn from Wikidata, a public knowledge base, it raised the precision of a Llama 65B model from 0.17 to 0.36. On biographies the FACTSCORE accuracy measure rose from 55.9 to 71.4 in the best variant.

What courts now require

The Fifth Circuit published a proposed rule that would have required lawyers to certify that no generative AI was used in a filing, or that a human had checked its text. After 13 public comments it declined to adopt it, and its notice says “I used AI” will not be an excuse for an otherwise sanctionable offense. On 18 February 2026, in Fletcher v. Experian, a panel led by Chief Judge Elrod fined the lawyer Heather Hersh $2,500 for a reply brief of which AI had drafted a substantial part, if not all, and which she did not verify.

The Supreme Court of Florida took a different route on 28 May 2026. From 15 June 2026 the signer of any filing represents that the legal authorities in it “exist and are accurately cited”, with sanctions available for a breach. Its administrative order bars lower courts from adding their own AI disclosure or certification rules, to end a patchwork of circuit orders.

India’s Supreme Court published draft Regulations for Use of Artificial Intelligence in Courts on 3 June 2026 and extended the comment deadline to 15 July. The draft requires a party who used an AI tool in preparing a pleading or evidence to file a declaration saying so, and says a person who files fabricated AI material cannot cite the AI as a defence.

In the US federal courts, Magistrate Judge Patricia Barksdale of Jacksonville, Florida, suggested on 29 May 2026 that Rule 11 gain a certification that cited authorities exist and are accurately cited. The staff memo in the agenda for the Advisory Committee on Civil Rules’ 21 October 2026 meeting recommends dropping the suggestion because current Rule 11 appears sufficient. The agenda book predates the meeting and records no vote. Its draft minutes show that in April 2026 the committee voted unanimously to drop earlier proposals on AI hallucinations, while recognising that AI may return to the agenda.