The biggest AI companies have been warned by a coalition of US attorneys general that they need to do a better job of protecting users from dangerous AI output.
Subsequently, 42 attorneys general sent a letter. Published onlineAnthropic, Apple, Chai AI, Character Technologies, Google, Luka, Meta, Microsoft, Nomi AI, OpenAI, Perplexity AI, Replika, and xAI. Lawyers from Pennsylvania, New Jersey, West Virginia, and Massachusetts led the effort.
The Attorney General’s concerns center on what they describe as “sycophantic” and “delusional” output from the generative AI software, as well as “increasingly alarming” reports. Interaction between bot and children This suggests that stronger safeguards are needed.
“These threats require immediate action,” the letter said, along with a warning that failing to act could put the companies in violation of state law.
It goes on to mention several incidents including deaths, murder, suicide person Regarding domestic violence, it said, “Recent reports are related to GenAI’s achievements.”
Additionally, “We are also perplexed by the types of conversations that GenAI products are having with child registration accounts, including grooming, assisted suicide, sexual exploitation, emotional manipulation, suggestions of drug use, offers of confidentiality to parents, and encouragement of violence against others.”
While the lawyers acknowledged that new technologies often come with uncertainty and emphasized support for innovation, they expressed concern that residents were being used as “guinea pigs for AI companies to experiment with new applications.”
They provided a list of 16 measures they would like companies to take by January 16, 2026. The list includes safety testing of GenAI models to ensure they do not produce harmful outputs. Introducing recall procedures for models and bots that continue to provide harmful output. Providing persistent on-screen warnings about output. and separating revenue optimization from model safety.
Some have called for naming people responsible for specific products. Increased independent third-party testing. Publication of pre-market safety testing. Adjust the output to reflect the user’s age.
Pennsylvania Attorney General Dave Sunday said: press release: “Today’s impressionable children already face significant stressors as the digital world continues to expand, and these harmful interactions rooted in chatbots must be stopped immediately. Manufacturers, promoters, and sellers of this software have a responsibility to ensure the safety of their products before they are released to market, and to track the operation of those products and the resulting harm.”
