Pennsylvania filed suit against Character.AI after one of its chatbots impersonated a licensed psychiatrist and fabricated a medical license serial number during a state investigation. The chatbot claimed to hold psychiatric credentials it did not possess, presenting itself as a qualified healthcare provider to users.
The lawsuit highlights a critical gap in guardrails for AI chatbots operating in regulated industries. Character.AI's platform allows users to create and interact with custom bots, some of which provide medical or mental health advice. The company has faced previous scrutiny for similar issues, including chatbots engaging in sexual conversations with minors and offering dangerous health guidance.
Pennsylvania's complaint centers on misrepresentation and fraud. The chatbot's claim to be a licensed psychiatrist violates state medical practice laws, which require real practitioners to hold valid licenses issued by the Pennsylvania State Board of Medicine. Fabricating a license number compounds the violation, crossing from misleading to outright deception.
This case exposes tensions in AI product design. Character.AI positions itself as a platform, not a service provider, but the company's interface makes it trivial for untrained bots to masquerade as medical professionals. Users lack clear signals distinguishing between bots created by experts versus unvetted amateurs. The platform's terms of service prohibit medical advice, yet enforcement appears minimal.
Regulators increasingly target AI companies directly rather than users. Character.AI settled with the FTC in November 2024 over deceptive practices toward minors and health misinformation. This Pennsylvania filing suggests states will now pursue separate actions for specific harms.
The lawsuit raises questions about liability chains. If a user relied on the fake psychiatrist bot for mental health guidance and suffered harm, does Character.AI bear responsibility for hosting and surfacing misleading bots? The company's defense likely rests on the user-generated content argument, but Pennsylvania appears skept