ChatGPT and the law collide in ways that have already cost attorneys money, cases, and reputations. Lawyers have been sanctioned for filing briefs built on invented citations. The American Bar Association has issued formal guidance mapping existing ethics duties onto AI-assisted work. The Copyright Office has ruled that purely AI-generated material cannot be registered. Malpractice insurers are quietly writing AI exclusions into their policies. Whether you are a lawyer using AI to draft, a business deploying it, or a consumer relying on its output, the core message from every regulator and court so far is the same: the old rules still apply, and the person using the tool is the one on the hook.
Fabricated Citations and Court Sanctions
The most concrete legal danger of using ChatGPT is hallucination. The tool produces text that reads like a real case citation but points to an opinion that does not exist, sometimes with fake quotes and invented docket numbers.
In 2023, a federal judge in the Southern District of New York sanctioned two attorneys $5,000 after they filed a brief containing ChatGPT-generated citations to cases that were entirely made up.1Justia Law. Mata v. Avianca, Inc., No. 1:2022cv01461 – Document 54 The court also required the attorneys to notify each judge whose name had been falsely attached to a nonexistent opinion. In 2025, a Colorado federal court sanctioned two more attorneys for a brief with nearly thirty defective AI-generated citations.
The rule these attorneys ran into is Federal Rule of Civil Procedure 11. When a lawyer signs a court filing, they certify that the legal arguments are “warranted by existing law” and that factual claims “have evidentiary support,” and that the certification follows “an inquiry reasonable under the circumstances.”2Legal Information Institute. Rule 11 – Signing Pleadings, Motions, and Other Papers Pasting ChatGPT output into a brief without confirming the cases exist does not meet that standard. Sanctions can include monetary penalties, orders to pay opposing counsel’s fees, and nonmonetary directives.
Ethics Rules Lawyers Must Still Follow
In 2024 the American Bar Association issued Formal Opinion 512, its first ethics guidance on generative AI. It does not ban the technology. It applies existing Model Rules to AI-assisted work and puts the responsibility on the lawyer.3American Bar Association. ABA Issues First Ethics Guidance on a Lawyers Use of AI Tools
Four Model Rules are central:
- Competence (Rule 1.1) requires lawyers to understand the benefits and risks of the technology they use. You cannot delegate research to a tool you do not understand well enough to catch its errors.4American Bar Association. Rule 1.1 – Competence
- Confidentiality (Rule 1.6) requires protecting client information regardless of its source, which means evaluating whether an AI platform stores, shares, or trains on your inputs.5American Bar Association. Model Rules of Professional Conduct – Rule 1.6 – Confidentiality of Information
- Communication (Rule 1.4) directs lawyers to reasonably consult with clients about how AI factors into the representation.
- Reasonable Fees (Rule 1.5) allows billing for time spent prompting an AI tool and reviewing its output, but generally not for time spent learning the tool itself.
Model Rule 5.3, which requires lawyers to supervise nonlawyer assistants so that their conduct aligns with the lawyer’s professional obligations, was written for paralegals and clerks but applies to AI tools by extension.6American Bar Association. Rule 5.3 – Responsibilities Regarding Nonlawyer Assistance Filing AI output without review sits in the same place as filing a paralegal’s draft without reading it.
Confidentiality When You Paste Client Data Into AI
When a lawyer drops a client contract into ChatGPT for analysis, that data may not stay private. Under the default settings for consumer-tier products, the provider may use submitted content to improve its models. OpenAI’s data usage policy states that content submitted to ChatGPT’s free and individual paid tiers may be used for model training, while business and enterprise tiers are excluded from training by default.7OpenAI. Data Usage for Consumer Services FAQ A limited number of authorized personnel may also access user content for abuse investigations, support, legal matters, or model improvement.
That creates a direct conflict with Rule 1.6, which requires “reasonable efforts to prevent the inadvertent or unauthorized disclosure of, or unauthorized access to,” client information.5American Bar Association. Model Rules of Professional Conduct – Rule 1.6 – Confidentiality of Information A lawyer who feeds privileged documents into a consumer AI tool without understanding the provider’s data practices may be violating that duty. The practical minimum: use enterprise-tier products that contractually exclude training on user inputs, turn off training features where available, and avoid inputting anything that identifies the client or reveals litigation strategy.
Attorney-client privilege compounds the risk. If confidential communications enter an AI system accessible to third parties, a court could find the privilege waived. Strip identifiers before using AI tools, or keep privileged material out of them entirely.
Court Disclosure Rules for AI-Assisted Filings
A growing number of federal judges now require attorneys to say whether AI was used to prepare a filing. As of early 2025, two federal districts mandate AI disclosure across all cases, and individual judges in at least ten other districts have issued standing orders requiring a certification of accuracy or an affirmative statement about AI use. A handful of judges have banned AI-generated content in court documents outright.
These rules vary courtroom by courtroom. Before filing in an unfamiliar jurisdiction, check the local rules and the assigned judge’s standing orders. Undisclosed AI use, when a court discovers it later, tends to be treated as a credibility problem, not a paperwork oversight. The safest approach: verify every citation independently, disclose AI use where required, and treat AI output as a rough first draft.
Billing for AI-Assisted Work
AI creates a real problem for hourly billing. If a document review that once took ten hours now takes ten minutes, billing for ten hours is indefensible. Model Rule 1.5, Comment 5, prohibits lawyers from exploiting fee arrangements through “wasteful procedures.” Formal Opinion 512 addresses this directly: a lawyer may bill for the time actually spent prompting the AI and reviewing the output, but the fee must be reasonable in light of the work performed.
Some firms are shifting to flat-fee or value-based billing for AI-assisted tasks, on the theory that charging hourly for work an algorithm completed in seconds will draw scrutiny from clients and ethics boards. The upside for clients is that lawyers can handle more matters at lower cost, even if individual invoices shrink.
Copyright in AI-Generated Content
The U.S. Copyright Office has drawn a firm line: copyright requires human authorship, and material generated entirely by AI without meaningful human creative control is not eligible for registration.8U.S. Copyright Office. Works Containing Material Generated by Artificial Intelligence The D.C. Circuit affirmed that position in Thaler v. Perlmutter, holding that “the Copyright Act requires all work to be authored in the first instance by a human being” and rejecting an attempt to register an image created autonomously by an AI system.9U.S. Court of Appeals for the D.C. Circuit. Thaler v. Perlmutter
AI-assisted work can still qualify, but only in parts. In the Zarya of the Dawn decision, the Copyright Office granted protection to a graphic novel’s text and to the author’s creative selection and arrangement of images, while denying protection to the individual AI-generated images themselves.10U.S. Copyright Office. Zarya of the Dawn Registration Decision Applicants must disclose AI-generated content and exclude it from their claim.
The Office’s 2025 report on copyrightability closed a door many creators had hoped was open. It concluded that “given current generally available technology, prompts alone do not provide sufficient human control to make users of an AI system the authors of the output,” because prompts “essentially function as instructions that convey unprotectible ideas.”11U.S. Copyright Office. Copyright and Artificial Intelligence, Part 2 – Copyrightability Report To claim copyright in a work involving AI, a human author’s creative expression has to be perceptible in the output, through substantial modification or creative arrangement. Typing a detailed prompt and accepting what comes back will not get you a registration.
Who Is Liable When AI Gets It Wrong
AI cannot be sued for malpractice. If a lawyer relies on ChatGPT-generated research without checking it and the client is harmed, the lawyer faces malpractice exposure, because the duty of competence does not disappear when the work is outsourced to a machine. If a business deploys an AI chatbot that gives customers harmful advice, existing product liability and negligence theories point at the business.
AI developers disclaim liability in their terms of service and warn users not to rely on outputs for critical decisions. Whether those disclaimers hold depends on the facts; one buried in a terms-of-service agreement may not protect a developer whose product was marketed as reliable for a specific professional use.
Malpractice insurance is where this gets sharper. Some insurers have started adding AI-specific exclusions to professional liability policies, eliminating coverage for claims arising from AI use. Others impose reduced limits for AI-related claims or require documented due diligence on AI systems as a condition of coverage. At least one insurer has taken the position that “blindly accepting AI output” can trigger an intentional-acts exclusion, leaving the attorney personally liable with no insurance backstop. Any lawyer using AI in practice should read their current policy for AI-related provisions before assuming they are covered.
Consumer Protection and Discrimination Laws Still Apply
Federal law reaches AI-related harms through statutes already on the books. Section 5 of the Federal Trade Commission Act makes “unfair or deceptive acts or practices in or affecting commerce” unlawful.12Office of the Law Revision Counsel. 15 USC 45 – Unfair Methods of Competition Unlawful The FTC has applied that authority directly to AI, bringing enforcement actions against companies making deceptive claims about AI capabilities. The agency put it plainly during its 2024 enforcement sweep: “there is no AI exemption from the laws on the books.”13Federal Trade Commission. FTC Announces Crackdown on Deceptive AI Claims and Schemes
Employment discrimination law applies when AI is used in hiring, promotion, or performance evaluation. Title VII of the Civil Rights Act prohibits employment practices that discriminate based on race, color, religion, sex, or national origin.14U.S. Equal Employment Opportunity Commission. Title VII of the Civil Rights Act of 1964 The Americans with Disabilities Act requires employers to ensure that hiring technologies do not unfairly screen out qualified individuals with disabilities and to provide reasonable accommodations when AI-based assessments are used.15U.S. Department of Justice. Algorithms, Artificial Intelligence, and Disability Discrimination in Hiring Pointing at the algorithm does not shift the liability. If the tool produces biased results, the employer using it is responsible.
Where the Rules Are Headed
The European Union moved first with comprehensive AI legislation. The EU Artificial Intelligence Act, adopted in 2024, classifies AI systems by risk level and imposes escalating requirements for transparency, accountability, and human oversight.16EUR-Lex. Regulation (EU) 2024/1689 – Artificial Intelligence Act Companies deploying AI tools that interact with EU residents must comply regardless of where they are based.
The United States has no equivalent federal law, but the regulatory picture is not empty. Beyond FTC and EEOC enforcement, several states have enacted targeted AI legislation focused on specific use cases. Some require developers to disclose the datasets used to train their systems, including whether those datasets contain copyrighted material or personal information. Others require that consumers be told when they are interacting with AI rather than a human. State activity is accelerating, and disclosure obligations are likely to expand.
The through-line across every one of these regimes is worth holding onto. Existing law already covers most AI-related harms through consumer protection, employment discrimination, malpractice, and professional ethics frameworks. The question is not whether AI is regulated. It is whether the people using it understand that the old rules still apply.