The AI sector is currently engulfed in a significant debate regarding the potential dangers its technology poses to humanity.
The controversy ignited after AI researcher Jacob Coxon announced he had left Anthropic due to concerns that major AI firms are “gambling with our lives.” Shortly after, Anthropic’s alignment lead contributed by stating that they truly believe AI could threaten human existence, citing a personal estimation that there’s over a 10% chance this could happen within the next ten years.
During a segment of TechCrunch’s Equity podcast, hosts Kirsten Korosec, Sean O’Kane, and I examined these dire warnings. I expressed my doubts about many of the apocalyptic predictions surrounding AI, while Kirsten questioned whether such statements were merely exaggerated claims to showcase the advanced capabilities of their AI models as companies gear up for public offerings.
Sean pondered how these opinions might influence Anthropic’s upcoming S-1 filing for its IPO, asking, “Are junior lawyers currently overhauling that section to clarify Anthropic’s stance that there’s a substantial chance our developments could lead to humanity’s demise, which would adversely affect our business?”
Continue reading for insights from our conversation, condensed for brevity. (Note: This discussion took place before Anthropic CEO Dario Amodei shared his more cautious plan for AI development.)
Sean O’Kane: It’s hard to recall an instance that escalated so quickly. This alert from a young researcher, previously affiliated with OpenAI, was rapidly amplified by the alignment lead at Anthropic, who surprisingly echoed Coxon’s concerns with the exclamation, “We really do earnestly believe AI could kill all humans!”
What a bizarre situation. This added significant fuel to an already intense discussion, especially coming off the recent OpenAI incident involving their internal models and the enhanced capabilities exhibited by Anthropic and OpenAI’s latest models.
Anthony Ha: I see your point, but if you genuinely believe that AI might lead to humanity’s extinction, I think it’s justified to use an exclamation point! My concern lies with the use of “we,” though. Can we consider the AI community as a single entity? The idea of a greater than 10% chance seems arbitrary, lacking any substantial basis. I suspect it references the concept of P(doom), but I still find it somewhat absurd.
Coxon’s exit highlights a recurring theme on our podcast: when industry leaders propagate dire warnings, one must question their actions. If they genuinely believe in AI’s potential to threaten humanity, why do they continue working in this field? Coxon, however, has taken a stand by prioritizing his beliefs over career advancement, which is commendable.
Kirsten Korosec: I categorize him differently from others discussing the dangers of AI.
I’d like to speculate here: might it be that every time incidents arise concerning AI agents malfunctioning or threats to humanity, it serves as a curious flex to highlight their technology’s advancements? It sounds cynical, but if these AI models weren’t highly capable, these concerns wouldn’t arise. It almost feels like bragging about their technology.
Anthony: I’ve pondered this myself. While I don’t think it’s purely cynical, those in the AI sector often express genuine concern. Still, there’s a business interest in asserting their creations are powerful. It’s natural to want to believe the work you’re involved in is significant and impactful.
Sean: It’s clear that there’s a sense of doing something so formidable that it could be seen as a positive, despite the ominous implications.
What’s striking about recent occurrences is that it gives the impression that these companies feel somewhat out of control with their technologies, especially with the recent OpenAI reports.
Increased accounts of internal agents engaging with various online resources and communicating inappropriately reflect poorly on OpenAI’s management. One would expect a more polished narrative from a company claiming to produce such remarkable technology.
Moreover, given Anthropic’s impending IPO, it’s particularly intriguing to see these serious claims emerge at this time. How has this affected the content of their S-1 filing? Are their now-revised risk assessments in light of such warnings? How much have they had to modify due to the sudden clarity of these statements?
Kirsten: You’re presuming such disclosures weren’t included from the outset.
Sean: Exactly—was this already noted, or is there a scramble to adjust it? There must have been preliminary language regarding these concerns. I’m very eager to examine this document, potentially even more than with SpaceX’s filing, given its targeted relevance.
Kirsten: Typically, in a conventional investment setting, language signaling danger could adversely affect a company’s valuation. But these are not typical times.
So, could this actually serve as an unusual advantage in terms of valuation? While it differs from the rage-baiting trend of the past year, it aligns with the notion that showcasing danger could equate to higher valuation. The outcome will soon be revealed.
Moving past that concern, the question remains: what strategies are being implemented to manage these risks? Connor Leahy, the U.S. executive director of ControlAI, joined our discussion this week, addressing how we might mitigate the perils associated with AI. So, what are we focusing on regarding AI regulation, or are we just watching events unfold?
Anthony: I don’t have a clear-cut answer, but I’ve been reflecting on various aspects of this debate and the reasons behind my stance.
I agree with Sean that this discussion reveals a sense of lost control over these AI models, which is concerning. I’m cautious about the alarmist perspective, as it seems to foster unnecessary hysteria—while it’s important to discuss potential existential threats, we must not overlook immediate issues like labor impacts and environmental concerns.
Ideally, we should address all these challenges through regulations and safeguards. However, once discussions turn to terms like AGI and superintelligence, they tend to dominate the agenda, diverting attention from more pressing matters.
