Ever since around the time of Jacob Coxon's post, I can't shake the logic that these capabilities will keep growing, proliferate, find their way into the wrong hands, and plausibly lead to mass casualties, or even the optimization of humans out of existence. Whatever you make of Coxon [1], the Hugging Face incident [2], or the frontier labs’ motivations, the argument is what sticks with me. My p(doom) isn't 100%, but it's high enough that I worry about it.
Plenty of AI researchers, of course, feel the same. In AI Impacts' 2023 survey of 2,778 AI researchers, the median estimate was a 5% chance of outcomes as bad as human extinction, and between 38% and 51% put it at 10% or higher. That was three years ago. [3]
What I'm struggling with is that the world seems to be carrying on as normal, either talking around the threat or ignoring it altogether. This feeling is new to me. I’d never believed in any doomsday scenario. In fact, I always thought people who did were more than a little crazy. Now I’m one of them.
I'm still going out with friends, talking with family, saving for retirement. But sometimes it’s hard to deeply care about anything else, including most of what hits the front page. If you feel this too, how are you handling it? And if you think I've got this wrong, I'd honestly like to hear why. It’s not what I want to be feeling about the future.
1. https://techcrunch.com/2026/09/09/gambling-with-our-lives-anthropic-researcher-quits-warns-against-self-improving-ai/ 2. https://simonwillison.net/2026/Jul/22/openai-cyberattack/ 3. https://arxiv.org/abs/2401.02843
After 15 years in Cyber i can say it is really easy to cause mass casualties. OT is criminally indefended and can have insane impacts. There is just no benefit for anyone to do such things.
Its not a matter of people not being able to do it. So having more means to do it, i dont think will change that.
People are filled with negativity each day and fear someone wants to destroy all humans. The vast majority of people, even those who make mistakes and do bad things, do not actively want to harm.
its painfuly obvious that the problem is people experiencing massive amounts of fear, not actually those fears being real or not.
What does fear do, and why must we experience it then?
Because fear makes people look externally for security. And security pays $$$$. This is a big reason for large societies to come into existence. To get rid of primal fears.
It is a marketing mechanism older than time itself -_- to get people to hand over their riches based on some probabilities of perceived risk.. usually very low ones. Since they are usually low probabilities, rational people would mostly take the risk, so they get overstated to emotionally trigger people.
This was pretty standard behaviour during the Cold War. Growing up in England I was well aware that a nuclear war could happen at any time and that England had absolutely no defence. And so we carried on living.
It's modern day fire and brimstone. "Forgive us our sins and render your flocks unto us, for we are speaking to the natal god".
Look up Yudwowsky's "math pets", his desire to essentially build a cult of women with math PhD's around him to sleep with. It's textbook narcissism that is focused on the self, not the world around him. That is not someone who should be able to dictate public discourse.