On April 25, 2023, LoDuca submitted a sworn affidavit (“Affidavit”), annexing copies of eight of the nine court‑ordered cases. The Affidavit adds that counsel was “unable to locate” the Zicherman case, six of the cases “may not be inclusive of the entire opinions but only what is made available by online database,” and one opinion is “unpublished.” The Affidavit did not address counsel’s reliance on those cases for arguments made in the Affirmation or seek to correct the Affirmation.
On May 4, 2023, the court issued an Order that LoDuca show cause on June 8, 2023, “why he ought not to be sanctioned pursuant to: (1) Rule 11(b)(2) & (c), Fed. R. Civ. P., (2) 28 U.S.C. § 1927, and (3) the inherent power of the Court” for citing non-existent cases in his Affirmation and submitting to the court his Affidavit with “copies of non-existent judicial opinions.” The Show Cause Order observed that “[s]ix of the submitted cases appear to be bogus judicial decisions with bogus quotes and bogus internal citations.” On May 26, 2023, the court issued a supplemental Order to Show Cause, incorporating its Order of May 4, 2023, and requiring LoDuca, Schwartz, and the Levidow firm to show cause why each should not be sanctioned pursuant to Rule 11.
During the hearing on the Order to Show Cause (“Sanctions Hearing”), plaintiff ’s attorneys insisted the citations were a mistake and denied any willful attempt to mislead the court. On the other hand, statements made by LoDuca and Schwartz at the Sanctions Hearing disclosed that they misunderstood how ChatGPT produced its outputs and had not verified the accuracy of each ChaptGPT output before using it as their own work. They also didn’t take the opportunity given them by the Reply Memo or Case Orders to correct their misplaced reliance on the ChatGPT output.
- Asked how he satisfied himself that the research reflected in the Affirmation was “accurate and truthful,” LoDuca admitted he didn’t. He relied on his colleague, Schwartz, to research and prepare it.
- LoDuca didn’t read any of the cases cited in the Affirmation.
- He did nothing to “ensure that those cases existed.”
- He didn’t read the five-page Reply Memo, which flagged cases that could not be located or did not stand for the propositions for which they were cited in the Affirmation.
- LoDuca “didn’t do anything other than” turn the Case Orders over to Schwartz, read the draft Affidavit, and note “the cases that were attached to it.”
- LoDuca didn’t notice the cases’ conspicuous textual incongruities, but acknowledged them when the court pointed them out.
The court’s examination of Schwartz revealed he didn’t really understand what the technology was doing; Schwartz assumed it was a “super search engine.” He did not appreciate that ChatGPT, a generative AI application, composes responses based on its predictive models and training.
- Schwartz’s first prompt directed ChatGPT not to “search, find, and retrieve” (as one might with a natural language search engine), but to “argue that the statute of limitations is tolled by bankruptcy of defendant pursuant to montreal convention.”
- His second prompt directed ChatGPT to “provide” cases that supported its argument: “User: provide case law in support that statute of limitations is tolled by bankruptcy of defendant under montreal convention.” In response, ChatGPT “produced” case names, citations, and a sentence ostensibly describing the holding.
- None of Schwartz’s case prompts directed ChatGPT to shepherdize or check if the “produced” cases had been distinguished, reversed, or overturned. The history of his prompts (included in Exhibit A to his Declaration) shows no curiosity about the existence of cases to the contrary.
- None of Schwartz’s prompts to ChatGPT reflected an awareness that the Montreal Convention’s limit on the time to bring an action had been interpreted by the Second Circuit to be a “condition precedent,” which could not be tolled by a bankruptcy filing, rather than a “statute of limitations,” which could be tolled by a bankruptcy filing.
- To comply with the court’s Case Orders, Schwartz “went back to the only place” that he could find the cases, ChatGPT, and prompted it to “produce” them.
On June 22, 2023, the court issued its opinion and order on sanctions, holding that LoDuca, Schwartz, and the Levidow firm “abandoned their responsibilities [to ensure the accuracy of their findings] when they submitted non-existent judicial opinions with fake quotes and citations created by . . . ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question.” The court found LoDuca and Schwartz “acted with subjective bad faith in violating Rule 11” and found the Levidow Firm “jointly and severally liable” for their Rule 11 violations.
The court sanctioned LoDuca, Schwartz, and the Levidow Firm under Rule 11 and, alternatively, under the court’s inherent power. The court imposed a $5,000 penalty and ordered the lawyers and law firm to send to their client, Mata, and to “each judge falsely identified as the author of the fake” opinion a copy of the court’s Sanctions Opinion and Order, a transcript of the Sanctions Hearing, and a copy of the April 25 Affirmation, including the “fake ‘opinion’ attributed to the recipient judge.”
ChatGPT and other generative AI applications might, with improvements, be determined by counsel to be appropriate for client work, provided counsel bases that determination on a careful examination of its capabilities and known faults and takes measures to avert or mitigate the adverse effects of those faults. Responsible introduction of any new technology by a lawyer or law firm for client work would include, at a minimum, adherence to procedures that determine the suitability of the tool for its intended purpose and manage limitations and faults. For generative AI, those basic steps would include:
- Undertaking a thorough due diligence check on the generative AI application to determine how the application could advance work done for a client consistent with counsel’s professional obligations;
- Reading key documentation on the developer’s website that describes the application, how the application works, what it was designed to do, its current limitations and deficiencies, and any tendency to put at risk a client’s interests, such as by generating inaccurate outputs. Such review should include, at a minimum, the developer’s Terms of Use, Frequently or Commonly Asked Questions (“CAQs”), and any System Cards, which explain, for example, a generative AI’s performance, capabilities, and limitations;
- Seeking out reviews and undertaking tests to identify inaccurate or biased results and training users about known indicia of inaccurate and biased outputs to help users recognize and remediate those deficiencies;
- Examining how use of the technology could pose security risks, including collection and processing of data that could compromise client confidentiality, inadvertently waive attorney-client or attorney work product privileges, and compromise the enforceability of client trade secrets and opportunity to file timely patent applications. If risks to client confidentiality and interests cannot be averted, the AI application has no place in a lawyer’s tool kit for legal work; and
- Considering carefully whether introduction of the technology might cause counsel to violate ethical obligations to (i) provide competent representation to clients, (ii) inform clients and obtain their consent to the use, (iii) protect client confidential information, and (iv) avoid unauthorized practice.
Terms of Use are routinely bypassed as users click to get to their online objectives. Enterprises assume the risk of missing valuable information and product warnings when their personnel do that. Consider what the documentation posted on OpenAI’s website reveals, for example, about the capabilities and faults in ChatGPT:
- Terms of Use, Section 3(d) (“Accuracy”):
- “Given the probabilistic nature of machine learning, use of our Services [ChatGPT] may in some situations result in incorrect Output that does not accurately reflect real people, places, or facts. You should evaluate the accuracy of any Output as appropriate for your use case, including by using human review of the Output.”
- Takeaway: Users must determine whether Output is accurate. Lawyers will need time to validate the Output (or risk Rule 11 sanctions).
- Terms of Use, Section 5(b):
- “You must implement reasonable and appropriate measures designed to help secure your access to and use of the Services. If you discover any vulnerabilities or breaches related to your use of the Services, you must promptly contact OpenAI and provide details of the vulnerability or breach.”
- Takeaway: OpenAI gives no assurance that ChatGPT will be cyber secure to use, and instead puts users on notice that “they are responsible for cyber risks and may have incomplete information about risks to their data and systems.”
- CAQs:
- “5. Who can view my conversations?
- . . . [W]e review conversations to improve our systems and to ensure the content complies with our policies and safety requirements.
- 6. Will you use my conversations for training?
- Yes. Your conversations may be reviewed by our AI trainers to improve our systems.”
- Takeaway: Client confidential information entered into a ChatGPT prompt should, upon transmission, be presumed to no longer be confidential.
- GPT-4 System Card:
- ChatGPT “maintains a tendency to make up facts, to double-down on incorrect information, and to perform tasks incorrectly. Further, it often exhibits these tendencies in ways that are more convincing and believable than earlier GPT models (e.g., due to authoritative tone . . . ), increasing the risk of overreliance. Overreliance occurs when users excessively trust and depend on the model, potentially leading to unnoticed mistakes and inadequate oversight. This can happen in various ways: users may not be vigilant for errors due to trust in the model; . . . or they may utilize the model in domains where they lack expertise, making it difficult to identify mistakes. . . . As mistakes become harder for the average human user to detect and general trust in the model grows, users are less likely to challenge or verify the model’s responses.”
- Takeaway: The diligence required to scrutinize, identify, and correct every inaccuracy in ChatGPT’s outputs suggests it will seldom reduce counsel’s work or time.
Beyond good practices for introducing new technology, lawyers must also watch for new rules from courts about use of AI. Judges in several jurisdictions have issued rules or practice directions prohibiting and requiring certification of non-use of generative AI or requiring a detailed disclosure of the use of generative AI applications in counsel’s court submissions.
III. Ill-Informed Technology Introduction Leading to Regulatory Enforcement Actions
Two settlements in cybersecurity and privacy cases this year illustrate how inadequate attention to foreseeable faults with new technology introduction led to consumer harm and regulatory enforcement action. These cases also foreshadow regulatory consequences for enterprises that rush AI applications to market leaving security, privacy, and other foreseeable dangers by the wayside. While the Ring and bitFlyer cases noted below are not about AI, they involve handling sensitive data and implementing effective risk management practices. Those activities are highly relevant to AI companies, which handle huge stores of data and are innovating at a furious pace in environments where risks are not completely known or understood.
A. FTC v. Ring LLC
The Federal Trade Commission (“FTC”) and Ring LLC (“Ring”), maker of security cameras for indoor and outdoor household use, agreed to a Proposed Stipulated Order settling FTC allegations of unfair and deceptive trade practices (the “Proposed Order”). In a complaint filed with the Proposed Order (the “Complaint”), the FTC describes a young technology company launching its indoor security camera business without establishing technical controls, procedures, or monitoring designed to limit employee and contractor access to video captured on installed devices. Customers installed the cameras throughout their homes, monitoring private spaces including bedrooms and bathrooms.
In the absence of security controls and training, some employees accessed and viewed highly sensitive video for illegitimate purposes. According to the Complaint, Ring responded to incidents incrementally, taking almost two years to narrow video data access by employees and overseas contractors on a “need-to-know” basis and even longer to institute privacy and security training for its employees. Furthermore, Ring’s early terms of use did not give Ring adequate customer consent to use sensitive video data for product development and improvement.
The FTC also alleged Ring’s pursuit of commercial success prioritized easy data access for its developers over security controls. “Despite promising greater security as its products’ core feature, Ring ignored information security considerations when management believed they would interfere with growth. In pursuit of rapid product development, before September 2017, Ring did not limit access to customers’ video data to employees who needed the access to perform their job function (e.g., customer support, improvement of that product, etc.).” What motivated Ring to change? Apparently, a different phase in its pursuit of commercial success; the FTC says Ring started “improving security practices to make Ring more appealing to potential acquirers.”
In the Proposed Order, Ring agreed to pay a money judgment of $5,800,000. In addition, Ring must delete video recordings collected without customer consents adequate to support Ring’s continued use. Going beyond the video data, Ring must delete work product—its models and algorithms—“developed in whole or in part from review and annotation of [the early recordings].” For twenty years, Ring must maintain a comprehensive privacy and data security program (“Program”), train its employees to safeguard the privacy, security, confidentiality, and integrity of customer sensitive data, and periodically test, monitor, reinforce, and certify the effectiveness of the Program.
The Ring case illustrates adverse consequences that may result from adopting a business strategy that pursues short-term profit at the expense of safeguarding customers’ sensitive data. Having failed to implement sound practices for cybersecurity and handling sensitive information, Ring had “no idea how many instances of inappropriate access to customers’ sensitive video data actually occurred.” Enterprises sometimes defer cybersecurity and privacy measures in the belief that it’s better to err and ask forgiveness than to invest time and funds in such measures from the outset. That choice may cost more than the amount of damages and fines, as illustrated by recent FTC cases that include forfeiture of digital assets created with misused data. That penalty can be a significant setback, depriving an enterprise of its early investments and, perhaps, its competitive advantages.
B. In re bitFlyer USA, Inc.
bitFlyer USA is a cryptocurrency trading platform licensed by the New York Department of Financial Services (“NYDFS”) under New York’s Virtual Currency Regulation and subject to New York’s Cybersecurity Regulation. bitFlyer USA had been a subsidiary of bitFlyer, Inc. ( Japan) until they and other affiliates were made wholly owned subsidiaries of bitFlyer Holdings, Inc. bitFlyer, Inc. ( Japan) supports bitFlyer USA with its internal operations including, among others, product and application development, information security and incident management, and information system development and maintenance.
Examinations by NYDFS staff identified deficiencies in bitFlyer USA’s cybersecurity program which led to an enforcement investigation. The Cybersecurity Regulation required bitFlyer USA to conduct periodic cybersecurity risk assessments according to practice standards that include written policies and procedures. A proper risk assessment is essential for a properly designed and managed cybersecurity risk program.
NYDFS found bitFlyer USA’s cybersecurity program lacking in its fundamentals. For example, bitFlyer USA adopted security policies copied from its Japanese affiliate without accurately or completely translating the policies to English, and it used template forms from various sources without even changing generic identifiers like “ABC Company.” bitFlyer USA’s policies and procedures were not customized to its needs and risks. bitFlyer USA also had not conducted a comprehensive cybersecurity assessment, and it did not have a written security policy approved by its board of directors.
The bitFlyer case cautions AI developers and adopters against cutting corners on security risk assessment, particularly in regulated industries where the law requires attention to security. Like other digital technologies, AI can have security vulnerabilities throughout its lifecycle. The AI life cycle is complex and multi-layered. It includes software, model, and algorithm design and development, collection of training data, incorporation in applications and business processes, execution of automated functions, and utilization of output for decision-making with varying degrees of human intervention. AI also poses risks its developers and the public do not fully understand or have not yet discovered. Knowing that, AI developers and adopters should conduct comprehensive risk assessments of AI applications from the outset and seek to implement safeguards, continuously monitor their effectiveness, and act promptly to respond to emergent risks and signs that safeguards might be insufficient. Failure to build effective security and risk programs alongside effective AI applications compounds potential adverse consequences for regulated activities like those in bitFlyer.
C. Nvidia’s Vulnerability
AI adopters must also be attuned to the allocation of risk for faults in AI applications. Consider this illustration. Nvidia computer chips are used by AI developers in building generative AI systems. When the AI research firm Robust Intelligence, Inc. breached safeguards in the Nvidia chips, Nvidia responded that it had fixed one of the root causes of the reported vulnerabilities. Nvidia’s VP of applied research was also quoted as saying Robust Intelligence “identified additional steps that would be needed to deploy a production application.” Lawyers: note in that statement the business strategy of shifting liability downstream; AI application developers will have to consider in their designs the vulnerabilities that may be introduced through the component chips they use. That type of risk‑shifting is accomplished by contract. In some cases, contracts could be Terms of Use accepted by a developer or tool user with a click.
D. Government AI Initiatives
AI-specific regulation emerged in multiple forms ranging from guidance for self‑regulation to legislation and judicial orders.
United States
- In October 2022, the White House released a Blueprint for AI Bill of Rights (the “Blueprint”). It sets out “five principles that should guide the design, use, and deployment of automated systems to protect the American public in the age of artificial intelligence”:
- (1) AI systems should be safe and effective;
- (2) algorithms should not discriminate and AI systems should be designed and used in equitable ways;
- (3) data practices should protect privacy;
- (4) design and use of AI should be disclosed and explained in plain language; and
- (5) there should be a human alternative and human backstop to AI applications.
- A Technical Companion offers practical steps for implementing the vision of the Blueprint.
- In January 2023, the National Institute of Standards and Technology issued a first edition AI Risk Management Framework. NIST intends the Framework to be used voluntarily “to improve the ability to incorporate trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems.”
- In February 2023, President Biden signed an Executive Order requiring AI systems designed, developed, acquired, and used by the federal government to operate “in a manner that advances equity.”
People’s Republic of China (“PRC”)
- In April 2023, the PRC released its draft Measures for the Management of Generative Artificial Intelligence Services (“draft Measures”) to be added to its existing regulatory regime for data security and privacy. The draft Measures would apply to “research, development, and use of products with generative AI functions and to the provision of services to the public within [PRC territory]. China’s regulations place responsibility for generative AI—from training to output—on AI providers. The regulations require providers to submit security assessments and algorithm information (an existing mandate) to the PRC government and to generate output that “reflect[s] the Socialist Core Values.”
European Union (“EU”)
- In May 2023, the European Parliament advanced the EU’s A.I. Act to regulate AI systems used in Europe. “The rules follow a risk-based approach and establish obligations for providers and users depending on the level of risk the AI can generate. AI systems with an unacceptable level of risk to people’s safety would be strictly prohibited.”
IV. Conclusion
As AI entered widespread use in generative AI applications during the year in review, the pace of enforcement and government initiatives to regulate AI also quickened. Judges in multiple jurisdictions responded to inaccurate court documents with new rules for counsel’s use of AI in court submissions. Governments advanced various initiatives to regulate AI, which diverge in their approaches and focus. Enforcement of existing privacy and security regulations identify deficiencies foreseeable in the AI space. These developments suggest enterprises and lawyers will need to do more than ever before to understand each new technology’s benefits and faults before relying on it to advance material business and professional objectives.
