AI

Computer Scientist Challenges Superintelligence Feasibility Amidst Industry Hype

A computer scientist argues that engineering bottlenecks, particularly data scarcity, make the promised artificial superintelligence unlikely, urging a focus on tangible risks over existential hype.

S
SciTech Editorial Team
Newsroom
··2 min read
Computer Scientist Challenges Superintelligence Feasibility Amidst Industry Hype
The recent public discourse surrounding artificial intelligence has been dominated by existential dread, sparked by high-profile statements from researchers at leading AI firms. When Anthropic researcher Jacob Coxon resigned, warning that his employer and OpenAI were "racing straight to self-improving superintelligence and gambling with our lives," the industry’s top figures validated these fears. Evan Hubinger, Anthropic’s head of alignment science, affirmed that the company earnestly believes AI could kill all humans, while CEO Dario Amodei echoed these sentiments. This narrative is rooted in the concept of recursive self-improvement (RSI), a decades-old theory suggesting that an AI capable of improving itself would trigger an accelerating cycle of autonomous upgrades, leading to artificial superintelligence (ASI) that exceeds human cognitive capabilities and evades control. However, a computer scientist recently challenged the engineering feasibility of this scenario in a doctoral seminar at Georgetown University. While the industry, including Anthropic and OpenAI, has shifted from theoretical speculation to practical engineering—citing internal data showing AI accelerating its own development—significant technical obstacles remain. The primary bottleneck is the availability of high-quality training data. Until 2023, progress in large language models (LLMs) was driven by scaling model size, from GPT-1’s 117 million parameters to GPT-4’s estimated trillion-plus parameters. Yet, post-GPT-4, scaling has yielded diminishing returns, forcing engineers to rely on "post-training" techniques like reinforcement learning, which require vast datasets of correct input-output pairs. This creates a "jagged frontier" where AI excels in structured fields like mathematics and coding, where data is easily generated, but struggles in unstructured domains like managing an email inbox. French AI researcher François Chollet hypothesizes that progress will plateau in areas bottlenecked by human-generated data. Consequently, the deployment of automated coding agents and LLM-powered researchers is unlikely to overcome this fundamental limitation. The scientist argues that while the promised intelligence explosion may never arrive, the pursuit of superintelligence itself carries the risk of creating real-world harms, urging a recalibration of public anxiety and regulatory focus toward these tangible engineering constraints rather than speculative existential threats.

Share

More from this section: AI