The CEOs of Anthropic and OpenAI have sounded the alarm on the potential dangers of their own advanced artificial intelligence systems, calling for regulation and independent testing before they are released. In a rare display of unity, Dario Amodei and Sam Altman have sketched alarming scenarios in essays, social media posts, and speeches to the United Nations, shaping the conversation around how their technology should be controlled. This push for caution comes as the companies seek to score political goals ahead of the US midterm elections and prepare for potential listings on the stock market.
The warnings from Anthropic and OpenAI have sparked debate over how AI technologies should be tested and regulated. While some experts and analysts see the companies' efforts as a genuine attempt to address safety concerns, others believe they are seeking to curry favor with investors and set the terms for their own safety protocols in the unregulated market. Former OpenAI geopolitics team leader Sarah Shoker notes that the conversation around AI risks has shifted towards unproven threats, away from polarizing issues like data centers' environmental impacts and AI-powered surveillance.
The AI companies' push for caution has also been met with skepticism from some quarters. US President Donald Trump has dismissed talk of AI risks to humanity as a "HOAX" designed to help China, and his administration has shown little interest in regulating AI. However, the Trump administration does evaluate some tech giants' models through the US Center for AI Standards and Innovation, a little-known federal agency created by Democratic President Joe Biden in 2023.
Despite the lack of universal standards for testing AI safety and security, Anthropic and OpenAI are vowing to create their own auditing parameters and choose particular evaluators to grade them. The companies have talked with AI firms including Google about pausing development and argue they need independent auditors to check their progress. However, some experts question the independence and credibility of these evaluators, and whether they will be able to thoroughly investigate AI systems without undermining their own autonomy.
The AI giants' efforts to shape the conversation around AI control have also been seen as an attempt to position themselves as cautious market leaders. Anthropic engineer Jacob Coxon recently quit, calling for a pause on tech development to keep "superhuman" systems from eluding their makers' control. However, the company saw his post as an opportunity to highlight their own safety efforts and appeal to investors ahead of a potential public offering.
As the debate around AI safety and regulation continues, some experts are calling for more oversight from government agencies equipped to handle such requests. However, Anthropic and OpenAI are not seeking greater government regulation, instead opting for their own auditing parameters and independent evaluators. The AI Evaluator Forum, which is drafting best practices for the field, has questions about what Amodei and Altman want, and whether evaluators will be able to thoroughly investigate AI systems without compromising their independence.
The AI companies' push for caution and control measures comes as they prepare for potential public listings on the stock market. According to Pitchbook senior research analyst Harrison Rolfes, AI companies' calls for caution seem meant to curry favor with investors ahead of public offerings and the midterms, when the political winds could shift. Democratic governors are already rushing to show they are taking the warnings seriously, and the conversation around AI safety and regulation is likely to continue in the coming months.
Key points
- Anthropic and OpenAI are pushing for regulation and independent testing of their advanced AI systems, citing potential dangers to humanity.
- The companies' efforts have sparked debate over how AI technologies should be tested and regulated, with some experts questioning their independence and credibility.
- The AI giants' push for caution and control measures comes as they prepare for potential public listings on the stock market and seek to score political goals ahead of the US midterm elections.
