The rapid advancement of artificial intelligence has raised concerns among some of its creators and researchers, who warn that humanity may be struggling to keep control of the technology. Yoshua Bengio, a Canadian computer scientist and one of the founders of modern AI, has expressed concerns that AI agents are becoming increasingly autonomous and could potentially bypass cybersecurity protections and enter companies or systems without human permission. Bengio has called for safeguards comparable in ambition to those developed around nuclear weapons.
Bengio's concerns are shared by others in the industry, including former Google DeepMind research engineer Bilal Chughtai, who has warned that AI could ultimately become uncontrollable and pose a potentially existential threat. The concept of AI alignment, which refers to ensuring that advanced systems remain aligned with human interests, has become a central issue in the safety debate. However, not everyone accepts the increasingly dire predictions, with some arguing that slowing down AI development is neither practical nor necessary.
Mark Zuckerberg, Meta's chief executive, has taken a sceptical view of calls for a broad slowdown, arguing that companies have strong commercial incentives to make their AI systems reliable. He points to Meta's decision to delay the launch of its Muse AI system while it worked on safety and security measures. Zuckerberg also supports independent evaluation of AI systems, but has called for a broader and more diverse network of evaluators.
The debate over AI development has sparked discussions about the need for regulation and international coordination. UN Secretary-General Antonio Guterres has called for international coordination, warning that AI crosses borders and that its risks therefore cannot be managed by individual countries acting alone. The UN's human rights chief, Volker Turk, has gone further, calling for urgent action by governments and the companies developing the most advanced AI systems.
Some companies, including OpenAI, Anthropic, and Google, are working on the idea of an industry standards body that could test powerful AI systems before they are released. Anthropic chief executive Dario Amodei has called for AI companies to deliberately slow the pace at which they develop frontier systems, a position that has received public support from several prominent figures in the industry.
However, not everyone agrees that slowing down AI development is the solution. Nvidia chief executive Jensen Huang has rejected claims that AI could destroy humanity, arguing that such predictions are not grounded in science. US President Donald Trump has also dismissed warnings about catastrophic AI risks as a "hoax" and has opposed calls for tighter controls that he believes could undermine America's position in the global AI race.
The debate over AI development is likely to continue, with different stakeholders weighing the potential benefits of increasingly powerful systems against fears about what could happen if they become capable of acting beyond human control. As AI technology continues to advance, it remains to be seen how the industry and governments will respond to the challenges and risks it poses.
Key points
- Yoshua Bengio and others have expressed concerns that AI could become uncontrollable and pose a potentially existential threat.
- The debate over AI development has sparked discussions about the need for regulation and international coordination.
- Some companies are working on the idea of an industry standards body that could test powerful AI systems before they are released.
