AI godfathers warn of runaway ‘intelligence explosion’

OpenAI chief scientist also among authors of report on prospect of ‘most consequential technological development in history’

Two of the “godfathers” of modern AI and senior executives at OpenAI and Anthropic have warned governments to prepare for an AI “intelligence explosion”, which they say could be the most consequential technological development in history.

A report co-authored by the Nobel laureate Geoffrey Hinton and the Canadian computer scientist Yoshua Bengio, considered godfathers of modern AI for their work in the field, urges politicians to act now before there is runaway progress in the technology.

The authors say they are concerned about an “intelligence explosion”, which they describe as a “dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less”.

The paper, titled “What if automating AI R&D triggers an intelligence explosion”, is authored by more than 20 people including Hinton and Bengio. Other authors of the paper include Jack Clark, a co-founder of Anthropic, and Jakub Pachocki, the chief scientist at OpenAI.

Their concerns focus on the possibility of AIs being able to improve themselves without human intervention, a process known as “recursive self-improvement”. In the paper it is broadly referred to as automated AI research and development. Once AI systems reach expert-level capabilities at AI R&D, a single developer could run a workforce equivalent to “millions” of top human researchers, the paper says.

The authors say the automating of AI R&D is the most likelybuting to improving their own technology and the resulting improved systems can be rapidly deployed once built

“An intelligence explosion could be the most consequential technological development in human history, compressing years of progress into months or less, threatening human control over AI systems, and severely eroding checks on power within and between states, companies, and branches of government,” the paper says.

Urging governments to take action quickly, the authors say: “Once an intelligence explosion begins, the window for action may close.”

They recommended a trio of policy priorities: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; finding ways to constrain breakneck AI development; and preparing to adapt to an intelligence explosion.

Anthropic and OpenAI have both agreed to having independent evaluators assess their models after warning that AI development was reaching a critical pitch in terms of safety.

Anthropic says AI now produces 80% of its own code, while OpenAI uses autonomous AI agents in areas such as the training of new models.

“Preliminary evidence suggests that a software-driven intelligence explosion is possible,” the paper says, adding this would lead to the “extremely rapid development of highly capable or superhuman AI systems”.

While this could produce medical breakthroughs and technological leaps, it would bring a trio of risks: powerful systems could enable biological and cyber threats that develop faster than the measures needed to counter them; as humans become less involved in AI R&D they could lose the opportunity to control systems; states could use an intelligence explosion to convert a modest lead in areas such as cyberspace into a decisive one.

The report acknowledges these impacts are “uncertain”. For instance, a technological breakthrough could take time to implement given the need to set up supply chains for any special materials involved or to comply with regulations. AI systems could also accelerate the speed at which risks are mitigated, say the authors.

If such a breakthrough happened, the authors say, AI systems could rapidly eclipse human experts in most fields and “radically accelerate” technological progress. Preparing for this “should be an urgent priority, including at the highest levels of government leadership”.

They say governments should consider policies that address the potential for an intelligence explosion, including: requiring transparent progress reports on AI-related R&D, including embedding independent auditors in companies; limiting how fast an AI can improve in a certain time period; working with datacentres to enable pausing certain AI R&D projects; making sure automated AI R&D systems are fully isolated and cannot escape human control; creating “emergency response plans” for the various scenarios that could emerge from an intelligence explosion.

Despite some glitches shown by systems, such as disobeying instructions, R&D projects that would take humans months to carry out could be fully automated by AI by 2028. “Productivity gains from AI R&D automation have not yet reached the threshold needed to trigger an intelligence explosion, but gains from newer systems are likely approaching that threshold,” they said.

Explore more on these topicsReuse this content