How real is the extinction risk to humans?
When the talk is about human extinction, there is bound to be a lot of hysteria. And hysteria is what I see on X, or as Nvidia CEO Jensen Huang said, “nonsense.” That as well as a chance to use the extinction talk to push specific narratives and agendas. Can AI kill all humans in the next few years? Zero chance. Can it kill all humans this century? Zero chance again, although human stupidity will surely lead to something catastrophic using AI. I understand that line may come across a bit semantic, but there is a difference. AI, on its own, is not going to kill anyone.
Irrespective, as the week progressed, we did see the AI vs humans take a centre stage as a point of conversation. It started with Jacob appearing on various television channels, including on CNN. And the week ended with Jacob’s boss Dario Amodei appearing on CNN to talk about his essay in which he proposed that everyone should slow down AI development.
His looong appeal — oh he loves writing long essays — has gained support from OpenAI boss Sam Altman, xAI boss Elon Musk, and Gemini boss Demis Hassabis. Essentially, all big US AI companies are in favour of slowing “the pace at which we improve the capabilities of AI models.” Sam Altman, to show that he is serious about joining hands with Dario, said his company would melt all GPUs if that would help humans stay alive when faced with a murderous AI.
If you remember, Sam and Dario refused to hold hands when PM Modi tried to bring the two together on the stage at India AI Impact Summit. They are such sworn enemies, since the days when Dario left OpenAI to start his own company. Now, they are willing to do the unthinkable.
From all the talk that we have heard this week, it would seem that human extinction, or even a global-scale disaster that kills people, due to AI is something very real. And yet, the skeptic in me, that guy who is not in Silicon Valley but outside its bubble, can’t help feeling that there is something else going on here. This spirit of cooperation that Dario, Sam, Musk and Demis have suddenly discovered within themselves may not be about saving humans but saving their own AI companies.
What I am about to say next is a conjecture, just like the arguments about AI killing all humans. You read it with the same scepticism that I apply when I read something from Dario or Altman. But I feel that AI killing all humans, and then all top AI companies talking of slowing down, is all about creating a regulatory and operating environment that concentrates power for these companies.
In the last few months we have seen AI becoming a technology that Anthropic and OpenAI can no longer control in a way that gives them great business or strategic returns. The Chinese AI labs are releasing AI models that are almost as good as the current AI models from the US. Just last week, DeepSeek released an update to its V4.1 model, which is almost as good as the OpenAI 5.6 Sol and Claude 5 at a fraction of cost. Also, the Chinese models are open weight models and are finding favour with companies, including the US companies.
Asking Jacob about AI extinction risk is like asking your AC guy about climate change… Not saying it's necessarily uninteresting or wrong per se but let’s keep things in perspective.
Hugging Face CEO Clement Delangue
With pressure from Chinese labs mounting, with investors seeking returns on the money they have so far put in companies like OpenAI and Anthropic, and with social resistance growing against tech companies due to the impact of AI on jobs and wealth distribution, how do you take the control back? There are several options, but I feel this week AI giants decided that doom and extinction talk is the best way to go about it.
David Sachs, a perennial Silicon Valley insider and who was AI Czar at White House earlier, was blunt in assessing the position that Dario, Sam and others took this week. On Sunday he wrote his own long note on X. His argument is that OpenAI and Anthropic neither need anyone’s permission nor the public spectacle they are creating to go slow on AI progress. “Go ahead and pace the frontier. You are the ones setting it. The easiest way not to build superintelligence is for you to agree not to build it,” he wrote.
His main thrust of the argument is that OpenAI and Anthropic are creating a mass hysteria so that they can get regulatory approval to consolidate their hold on the market.
"Stop pretending the motivation to slow down is purely altruistic,” wrote Sacks. “Stop pretending antitrust law has to be suspended so you can form a cartel. Stop pretending you need a regulatory approval process that supersedes product liability. Stop pretending you need those (AI) evaluators to police competitors who aren’t even at the frontier.”
According to Sacks as well as many others who have made similar arguments in the last week, what Jacob started and then used by Sam, Dario and others to articulate their own position, is nothing but a campaign to:
— Get favourable regulations that put OpenAI, Anthropic and other US tech giants in position to dictate the AI market.
— Get a regulatory environment that kills or throttles open-source competition, most notably from China.
— Get a regulatory environment that shields them from liability in case of rogue AI creating some damage, such as hacking a crucial part of digital infrastructure.
And I agree with it all. Extinction, of humans or otherwise, is not a word that needs to be thrown around lightly. And if tech bros, even if they are from Anthropic, are throwing this word so casually, I am not going to take it seriously. Because, as Hugging Face CEO Clement Delangue wrote earlier in the week, “asking Jacob about AI extinction risk is like asking your AC guy about climate change… Not saying it's necessarily uninteresting or wrong per se but let’s keep things in perspective.”
