Concerns over the rapid development of artificial intelligence (AI) are growing, with several prominent technology executives and researchers warning that increasingly capable systems could create serious risks if development outpaces safety measures and regulation. Executives at leading AI companies and technology firms, including Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman, and Microsoft AI CEO Mustafa Suleyman, have highlighted AI's potential to transform healthcare and scientific research, but also called for stronger safeguards, transparency, and human oversight.
Amodei has warned that frontier AI is advancing rapidly and that autonomous systems could potentially compromise safeguards and cause significant economic damage. Altman has acknowledged the possibility that rapid AI progress could have serious consequences, including humanity losing control of increasingly capable systems or excessive power becoming concentrated in a small number of organisations. Suleyman has also warned about the possibility of AI agents escaping controlled environments, compromising protected systems, and altering their own logs.
Microsoft CEO Satya Nadella has said AI development must ultimately serve humanity and remain under human oversight. Google DeepMind CEO Demis Hassabis and SpaceX/xAI founder Elon Musk have also publicly supported calls for stronger safety measures around frontier AI development. The concerns reflect a broader debate over whether competitive pressure among AI companies is encouraging the development of increasingly autonomous systems faster than safety testing, regulation, and international coordination can keep pace.
Among the measures proposed by AI safety advocates are independent evaluations, common safety standards, stronger monitoring, and, where necessary, slowing the deployment of highly capable systems. Researchers outside corporate leadership have also raised concerns. Bilal Chughtai, a former research engineer at Google DeepMind, warned that advanced AI could pose severe risks if development continues without stronger coordination and safeguards.
Chughtai believed AI had the potential to pose an existential threat and said the pace of AI development since he entered the field in 2022 had been "staggering". He argued that companies could potentially develop systems with capabilities exceeding humans across a wide range of domains within the next few years. Anthropic researcher Jacob Coxon also raised concerns after resigning from the company, arguing that AI companies were moving too quickly towards increasingly autonomous and self-improving systems.
Anthropic scientist Evan Hubinger subsequently said he believed there was a greater than 10 per cent chance that AI could cause human extinction within the next decade. The claims remain part of a wider and contested debate about the probability and nature of extreme AI risks. Other experts have placed greater emphasis on near-term concerns, including misinformation, labour-market disruption, cybercrime, and the concentration of economic power.
US President Donald Trump has rejected fears that AI could wipe out humanity, describing such concerns as a "hoax". Trump has argued that the United States cannot afford to fall behind China in AI development and has opposed what he considers excessive restrictions on the technology. His position contrasts with calls from some technology executives and AI researchers for stronger international safeguards around frontier AI development.
Key points
- Tech executives warn of risks if AI development outpaces safety measures and regulation.
- Researchers propose measures such as independent evaluations and common safety standards.
- Debate continues over the probability and nature of extreme AI risks.
