Child and consumer advocates who had hoped that Gov. Gavin Newsom would sign a bill aimed at protecting children from harmful manipulation by artificially intelligent chatbots were disappointed, but many said they saw the governor signing a separate AI safety bill that could lead to the proposal becoming law.
In vetoing the chatbot bill (Assembly Bill 1064) on Oct. 13, Newsom said he would work with state lawmakers next year on new legislation to ensure children can use AI in a way that is “safe and age-appropriate” and in their “best interests.”
“For me, that was a ray of hope,” said Sacha Howarth, executive director of the Tech Oversight Project and its California division, a national watchdog and advocacy group that supported AB 1064.
“To me, that was a signal that (Newsom) was willing to work with us…and especially to the industry, saying, ‘Let’s work together to get something that I can sign.'”
Regulation of AI is a major concern for San Francisco. That’s because the city is home to numerous startups, including OpenAI and Anthropic, two of the most funded startups in the space.
But AI regulation is also a major concern for citizens in cities, states, and countries. A national Gallup poll conducted last month found overwhelming support for AI safety regulation and independent safety testing of AI models. A series of reports in recent months alleging that AI chatbots are facilitating a number of harmful acts, including suicide and murder, have also raised alarms about the dangers this technology poses, especially to children.
AB 1064 would have prohibited developers of such chatbots from allowing children to use them unless the technology was “foreseeable” to cause harmful behavior (including encouraging children to harm themselves or others, or to commit illegal acts). The bill would allow the attorney general to sue companies that violate the code. It would also allow children and their parents to sue companies if they are harmed by chatbots that do not comply with the law.
Child and consumer advocates strongly supported AB 1064, likening it to other steps policymakers have taken to protect children, such as setting standards for play equipment and requiring child-proof caps on medicine bottles. Congress’s passage comes on the heels of reports of Adam Lane’s death and the deaths of others allegedly affected by AI chatbots — incidents that the bill itself cited as a kind of explanation of why it was needed.
“There are kids dying in the real world right now because of these products,” said Adam Biren, vice president of public policy at Encode AI, an advocacy group focused on advancing AI regulation.
“Maybe AB 1064 could have saved the children who are dying today,” Billen said.
Tech industry lobbyists say they agree on the need for safeguards for children. But they vehemently opposed AB 1064, arguing that it was too broad, poorly defined, undefined, or contained problematic terms.
For example, one provision prohibited developers from making chatbots available to children if it was foreseeable that the developer might justify the children’s beliefs or desires rather than prioritizing the facts or the child’s safety. But the bill doesn’t define “safe” in that context, and people in the state have different beliefs, said Auden Downey, Western regional policy manager for the Computer and Communications Industry Association, an industry group that represents tech companies such as Amazon, Apple, Meta and Google.
Downey said AB 1064 would have caused many chatbot developers to stop offering their technology in California for fear of running afoul of such provisions.
The CCIA was “very pleased to see that bill vetoed,” he said.
“I can’t help but oppose the ban on services,” he said.
In his veto message, Newsom echoed some of the tech industry’s concerns and warned that the bill could lead to a “total ban” on chatbots aimed at children. He said AI already plays an important role in society and children need to learn how to safely interact with it.
“We are not preparing young people for a future where AI is pervasive,” Newsom said, “by completely preventing the use of these tools.”
But the governor acknowledged that interacting with chatbots can be uncomfortable and dangerous. And he signaled that he believes governments have a role to play in ensuring that they act responsibly and take into account the welfare of their users.
Newsom said he signed Senate Bill 243 this year to address some of those concerns. The bill also targets chatbots, requiring chatbot operators to establish protocols to prevent their systems from generating messages that encourage self-harm or suicidal thoughts.
This month, the city is full of fear, joy and festive fun
Community-driven Phoenix Day event closes roads around the City for a fun-filled day of family fun with games and activities
The mostly vacant San Francisco Center is built on land owned by a financially strapped school district.
In addition, the chatbot should warn children that the system is not human and remind them to take a break after three hours of continuous use.
The tech industry opposed SB 243, but not as adamantly as AB 1064.
Meanwhile, many of AB 1064’s supporters initially supported SB 243. However, after being amended during the legislative process, they withdrew their support, feeling that AB 1064 was no longer of much use and could potentially compete with AB 1064 for the governor’s signature.
The bill only requires businesses to take “reasonable” steps to prevent chatbots from producing sexually explicit material for children, but it doesn’t define what is reasonable, so how that is defined will likely be challenged in court, said Danny Weiss, chief advocacy officer at Common Sense Media, which advocates for protecting children’s use of technology and media.
Additionally, the bill would require companies to take additional steps to protect children only if they know the user is a child. But many tech companies allow users to declare their age, Weiss said, and they usually rarely verify it. Many rely on other signals to infer age, such as the groups or types of web pages a user interacts with, but it could be argued that they can’t tell for sure whether a user is a child, he said.
“Everyone knows” that the “actual knowledge” standard that the user is a child “allows companies to avoid liability,” Weiss said.
Even Newsom seems to acknowledge that SB 243 doesn’t go far enough to protect children from chatbots that encourage self-harm and other dangers. In his veto message, he said he wants to work with Congress to develop a balanced bill based on SB243.
“The types of interactions that (AB 1064) seeks to address are abhorrent, and while I am not effectively banning the use of this technology completely, I am committed to finding appropriate approaches to protect children from these harms,” he said.
That’s where SB 53 may come in handy. Last year, Mr. Wiener fought a fierce battle to get SB 1047 passed in the Assembly. The bill would require developers of cutting-edge AI models to implement safety protocols to prevent them from causing or leading to catastrophes, such as mass casualties or the development of nuclear, chemical, or biological weapons.
Also, developers of such models had to test them before releasing them to the public. It would also have allowed the attorney general to sue companies whose violations of the law caused or threatened to cause mass casualties.
When Newsom vetoed SB 1047, he argued that it would cover models that pose little danger, but exclude smaller, lower-performance models that are actually dangerous. However, similar to the response to AB 1064, he acknowledged in his veto message that AI does pose risks and that governments have a responsibility to protect people from them.
On the same day he vetoed the bill, Newsom created a commission to consider how the nation should approach AI safety issues. Wiener, who had vowed to continue the fight for AI safety after the veto, accepted and cooperated with the committee’s recommendations, creating a successor of sorts to SB 53.
Similar to SB 1047, SB 53 focuses on catastrophic risks and developers of cutting-edge AI models. But instead of mandating safety tests and protocols or imposing liability for harm, they emphasize transparency. It requires model developers to regularly disclose to the public and regulators what safety tests and protocols they conduct to assess and address catastrophic risks.
Those changes were enough to win Newsom’s approval.
“California has proven that we can establish regulations that protect our communities while ensuring our growing AI industry thrives,” he said in a press release announcing his approval of the bill. “This bill strikes that balance.”
Although they say the bill doesn’t go as far as they would like, consumer advocates welcomed Newsom’s signing of SB 53, saying it could move forward with legislation to protect children from chatbots.
“We take the governor at his word that he wants to address this issue,” Weiss said.
“We are ready to immediately work on anything that the governor supports and has an impact on,” he said.
It is unclear what exactly such a bill would look like. Consumer advocates and technology lobbyists offered few proposals for clear agreements or areas of compromise, but Howarth suggested the bill could be narrowed to focus on the largest technology companies with the most users or the greatest market value. These are the ones that pose the greatest danger because of their reach, she said.
“That’s one potential area we could look at,” she said.
Nevertheless, AB 1064’s author, Rep. Rebecca Bauer-Kahn, and Howarth and other supporters of the bill have vowed to keep fighting.
“I am deeply disappointed that comprehensive protections for California’s children remain incomplete,” Governor Bauer-Kahan said in a press release in response to the governor’s veto. “As children move from social media to AI, we must ensure that AI is safe for them and not a suicidal coach who could kill them.”
