A top Wall Street law firm used AI to draft a court filing and it completely made up facts

A respected Wall Street law firm has stumbled into the growing debate over artificial intelligence in professional services after admitting it submitted a court filing full of false information generated by an AI tool. Sullivan & Cromwell, known for representing major financial institutions and corporate giants, told a federal judge in Manhattan this week that a recent document filed in a bankruptcy case contained numerous inaccuracies-some of which never existed outside the AI's output.

The filing, related to the restructuring of the Prince Group, included citations to non-existent court rulings and mischaracterized legal precedents. When inconsistencies were flagged, the firm traced them back to an AI assistant used during the drafting process. The tool, designed to speed up legal research and document preparation, had 'hallucinated'-a term used when AI confidently generates false or fabricated content.

In a letter to Judge Loretta Preska, partners at Sullivan & Cromwell apologized and took full responsibility. They described the incident as a breakdown in oversight and said they are launching an internal review of how AI tools are used across the firm. Some of the partners involved bill more than $2,000 per hour, underscoring the high stakes of even a single error in such elite legal work.

The case has drawn sharp attention from legal ethics experts and tech watchdogs alike. While many law firms have quietly adopted AI for routine tasks, this is one of the first confirmed instances where hallucinated content made it into an official court submission by a top-tier firm. The slip-up raises concerns about accountability, especially when AI is used in time-sensitive or complex litigation.

Legal professionals across the country are now re-examining their own AI policies. Some firms have reportedly paused the use of generative AI tools pending new guardrails. Others are calling for mandatory human verification of any AI-assisted work before it's filed with courts.

There's no indication that the firm intended to mislead the court. Instead, the episode highlights a broader tension: as AI becomes faster and more embedded in knowledge work, the risk of overreliance grows-even among the most experienced professionals.

Sullivan & Cromwell declined to name the specific AI tool used, citing confidentiality agreements. But the firm confirmed it is working with its technology providers to improve detection of hallucinated content. For now, the case serves as a cautionary tale about the limits of automation in fields where precision is non-negotiable.