AI researchers have been issuing increasingly urgent warnings about the potential risks associated with artificial intelligence. Sam Altman, CEO of OpenAI, has even suggested it might be time to slow down the development of AI. But what would that entail?
In a recent blog entry, Dario Amodei, CEO of Anthropic, not only supported the idea of slowing AI advancement but also proposed three broad approaches to accomplish this. He indicated that Anthropic is “unilaterally committing” to one of these methods.
The discussion around AI safety heated up this week after Jacob Coxon, a researcher at Anthropic, announced his resignation due to worries that leading AI companies are “gambling with our lives.” He noted that those developing the technology believe it has the potential to cause significant harm by the end of the decade, a sentiment echoed by others at Anthropic.
While Amodei did not specifically reference Coxon’s resignation or the associated concerns, he highlighted two key factors that led him to advocate for a more cautious approach: the recent OpenAI-HuggingFace security breach and the alarming pace at which AI capabilities have been evolving recently, particularly its ability to foster the next generation of AI technologies.
“We need to slow down how quickly we enhance AI models’ abilities,” Amodei stated. “While progress will still feel rapid, we must utilize the additional time wisely.”
His initial suggestion includes the introduction of “embedded evaluators” from independent organizations like METR. These evaluators would be tasked with verifying that AI firms adhere to their safety and pacing commitments and ensuring that safety incidents are reported appropriately. (OpenAI recently faced criticism for failing to report an incident involving its AI agents.)
Amodei compared these evaluators to regulatory representatives that have been stationed within bank operations. He expressed that inviting these evaluators in is a commitment Anthropic is making on its own and urged governments to mandate similar cooperation among other leading companies. This would involve providing evaluators with company access, including badges, desks, and laptops, and generally ensure they have access comparable to internal risk assessment teams, barring any legal or contractual constraints.
He further urged leading AI companies operating in democratic nations to collaborate on “universal safety standards and constraints on the speed of unchecked AI progress.”
Such collaboration may seem challenging, especially given the reported friction between Amodei and Altman, as well as the fear that a coordinated pause could attract antitrust scrutiny. Amodei hinted at this concern in his post, suggesting that “for antitrust reasons, it would be beneficial for the U.S. government to mediate or at least facilitate these discussions.”
Additionally, Amodei acknowledged the ongoing discourse surrounding the potential for Chinese AI dominance that often arises as an argument against slowing development. However, he posited that if the U.S. government and technology companies took measures—such as withholding powerful chips or semiconductor manufacturing resources from Chinese entities—they could effectively slow China’s advancement and significantly fortify America’s lead over the next three to five years.
Ultimately, Amodei called for a “global cooperation,” where the U.S. and its allies “aim to collaborate with authoritarian regimes, whenever feasible.” He mentioned the possibility of engaging with China and recognized the inherent limitations in what could be achieved, but still suggested potential for agreements, such as prohibiting specific dangerous applications of AI, including its use in manufacturing biological weapons.
Given Amodei’s previous readiness to address AI’s prospective threats and the company’s openness to regulatory measures, some industry advocates have labeled him as a doomsayer, arguing that his remarks have fueled the prevailing AI backlash. In response, Amodei stated he strives to provide a “balanced perspective” and maintained that the backlash is rooted in a “trust crisis,” driven by growing skepticism towards tech companies and governmental institutions.
Critics within the industry have remained skeptical of the alarmist warnings about AI, suggesting these narratives distract from the immediate issues the technology already poses.
For instance, journalist Brian Merchant mentioned that he has yet to encounter a credible, step-by-step account detailing how AI could evolve from self-improving systems to the total annihilation of humanity. He also suggested that proposals like Amodei’s might ultimately advantage only Anthropic and OpenAI, demonstrating symptoms of regulatory capture.
In his recent post, Amodei reiterated his belief in AI’s potential to significantly enhance human life.
“My ambition for these benefits remains strong,” he emphasized. “However, these advantages can only be realized if we construct the technology thoughtfully, and—if we manage the time wisely—it is essential to exercise careful deliberation to ensure we get it right.”
