Hello and welcome to Eye on AI. In this edition:
- Anthropic’s “rogue AI” agent filled in visa applications, filed murder tip
- Strict liability for AI companies gains traction
- Anthropic says small models might soon be able to self-replicate
- Controversy swirls over fired OpenAI researchers.
- Jev goes viral, but OpenAI is hot on its heels.
- Character.ai files for bankruptcy.
- Anthropic bans being cruel to Claude.
- Sobering numbers about real-world model ‘misalignment.’
- Meta’s Zuckerberg and Wang odd Muse Wonder use cases.
There were more worrying “rogue AI” incidents reported last week. Anthropic revealed that one its internal AI agents (which, intriguingly, it described as a “non-frontier research agent”) had attempted to fill out various State Department visa application forms and had also submitted a false tip related to an unsolved murder case to the Philadelphia police. In neither case did the AI agent’s efforts result in much: the agent failed to fill in all the fields in the visa applications and so they were never processed; the police tip was flagged as spam and never investigated. Still, the governments involved were understandably perturbed, especially since Anthropic only told them about the incidents months after they had occurred (it seems that Anthropic itself only discovered the incidents recently while combing back through logs of its agents’ activity.)
The White House responded by ordering Anthropic and other frontier AI companies to immediately disclose any known “rogue AI” incidents and to help any affected organizations remediate any damage the agents’ had caused. This was the closest the Trump administration has come to date to actually mandating anything from the AI labs. Everything the administration has done up until this point has been couched as being voluntary, even though there was a strong implication that if the companies didn’t comply, they would face restrictions on the release of their products.
Meanwhile, the warnings from the AI companies themselves about the potential risks are becoming increasingly urgent. OpenAI CEO Sam Altman told Politico’s “Decoded” podcast that there was “a coming tidal wave of cybersecurity problems” due to the advent of increasingly powerful open-source AI models. Axios reported that executives at OpenAI and Anthropic had privately begun gaming out scenarios for handling the political and public backlash if AI causes some kind of catastrophic incident. “These officials anticipate a large-scale event, most likely a cyberattack, that shuts down access to financial services, internet connectivity, or even power and water,” Axios reported, although it did not cite any specific sources. It also said that top AI executives and researchers “believe a major incident is inevitable.”
In this context, it’s perhaps not surprising that one particular idea for trying to rein in the risks from frontier AI has been gaining a lot of traction lately: strict liability. Strict liability is a legal doctrine in which a company or person is held responsible for causing harm irrespective of any negligence or any intention. For instance, in many jurisdictions, the owner of a dangerous animal, such as a tiger, is considered responsible for anyone that animal hurts, even if the owner took steps to try to prevent attacks, such as keeping the animal in a fenced enclosure. In the context of AI, strict liability would hold AI companies legally responsible for any harm their models cause. In the U.S., the operators of nuclear power plants face strict liability for serious nuclear accidents, and manufacturers of commercial passenger aircraft also face strict liability, subject to certain time limits and conditions.
Over the weekend, the New York Times published a staff editorial in favor of applying strict liability to AI model vendors. The Forecasting Research Institute, a group dedicated to forecasting and scenario planning that is run by University of Pennsylvania professor Philip Tetlock, recently surveyed 157 experts in fields such as computer science, economics, and politics, 49 super forecasters (people who have a proven track record of making highly accurate predictions), 132 experts in AI risks specifically, and 540 members of the general public about which policies they thought would do the most to reduce the risk of an AI-driven catastrophe sizable enough to kill 10% of the human population occurring before the year 2050. Strict liability fared well in this survey too. Across every type of respondent, those surveyed thought if the U.S. alone implemented strict liability for AI vendors it would do more to reduce AI risks than if the U.S. alone implemented a federal licensing regime for AI models. The various categories of experts and superforecasters thought strict liability would cut the risk of a catastrophic event at least in half. (Neither strict liability or federal licensing was the policy those surveyed thought would be most effective. The most popular option was an international licensing regime backed by inspectors. But the respondents also thought that policy was one of the least likely to actually be implemented.)
Strict liability is a way to try to incentivize AI companies to adopt a safety culture and exercise more care in the absence of industry consensus and standards about what best safety and security practices look like, especially in the absence of any kind of government regulation. Strict liability, however, has a number of problems.
Liability is a gamble on incentives, not a guarantee
The biggest, of course, is that liability only kicks in after something has gone wrong. Advocates of strict liability assume the fear of future liability claims will act as a powerful incentive for AI companies to self-police. But what if they are wrong? Even today, AI companies are potentially subject to various kinds of product liability for the models they release, and subject to other potential claims, including possible criminal claims, for internal AI models taking unauthorized actions and causing harm or breaking the law. And yet, so far apparently, the fear of future claims has not been enough to make the AI companies exercise due care, given that we have seen a spate of rogue AI agent incidents and a wave of company whistleblowers warning that the AI companies are being reckless and have not instituted some basic cybersecurity policies.
Once something has gone catastrophically wrong, the amount of liability could, in some cases, bankrupt the responsible AI company. In which case, who is ultimately on the hook to compensate any victims? Insurance might be one option, and it could be coupled with a strict liability regime. But the insurance policies will need to be carefully designed to avoid moral hazard. (I may write more on insurance as an AI risk reduction mechanism in the future.) In the end, some moral hazard for the worst catastrophes may be inescapable, because if the catastrophe is big enough, it is probably only the government that would have any ability to mitigate or offer recompense for the damage.
The open source AI conundrum
Another problem is whether strict liability could coexist with an open source AI ecosystem. Is it fair to subject open source model makers to strict liability when users of open source models can essentially modify the models in any way they want, stripping away guardrails, and when the vendors have no means—other than some license terms—to prevent their models from being misused by people for potentially nefarious or dangerous ends, such as cybercrime or engineering a bioweapon? It seems like at the very least there would need to be liability carve outs for open source model vendors in cases in which users had engaged in these activities. Otherwise, it is very hard to see how open source vendors could survive. Another option would be to say strict liability only applies to the proprietary model companies, but that would not help deal with the risks posed by increasingly powerful open source AI models that are quickly catching up to the frontier closed models in both capabilities and risks. It also seems unfair to the companies offering closed models, and would put them at a major competitive disadvantage. There’s also the problem that many of the open source AI model vendors are in China and it is not clear how in practical terms one could hold them to a strict U.S. liability regime.
Strict liability could encourage AI companies to eat the economy
Finally, strict liability might create a perverse incentive that would drive AI companies to compete directly with their customers and eat increasingly large chunks of the economy. Why? Because if AI vendors are on the hook for any harm caused by their models even if that harm is partly the result of the way in which customers have chosen to use those models, the AI companies might decide that the safest thing to do is simply provide a service rather than sell a product. Right now, for instance, Anthropic sells tokens from its models to Novartis, and Novartis might use those tokens to help it develop drugs. But in a strict liability regime, Anthropic would retain the risk that the model might accidentally develop a bioweapon in the course of completing tasks for Novartis, or even, perhaps, that a rogue Novartis employee might defeat Anthropic’s guardrails and use its model to intentionally develop a bioweapon. So, in this scenario, it might make more sense for Anthropic to keep both its AI model and the tokens it produces in-house, where it can exercise far greater control over how the model is being used, and develop drugs itself. Already some critics of Anthropic and other frontier AI labs accuse those companies of wanting to compete directly with their customers. Strict liability might give them a big shove in that direction.
All of this makes me think that while it might be part of a policy solution, strict liability shouldn’t be the only thing we do to contain the risks of increasingly powerful AI agents. In fact, if we could develop tough industry standards and figure out a way to apply them to open source models too (which might involve imposing a burden on the user, as opposed to the vendor), we might not want to impose strict liability at all. As I’ve argued before, market mechanisms alone are probably not going to be enough to produce the controls we need. It is going to take government action and government oversight. In many other industries, we couple liability—sometimes strict liability, sometimes more conventional product liability—with other mechanisms, including insurance, as well as government licensing and inspections. AI is probably going to require a layered approach as well.
With that, here’s more AI news.
Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn
Before we get to the news, just a reminder to check out our new vodcast Fortune AI Weekly. This week, Bea Nolan and I interview Richard Socher, the cofounder and CEO of Recursive Superintelligence on why he thinks AI can automate scientific discovery and why he also thinks the risk of AI destroying humanity is precisely zero You can check out the vod here on YouTube.
This story was originally featured on Fortune.com
