I’ve been putting GLM 5.3 through its paces this week, poking at it the way I poke at any new model. One conversation kept dragging me back to a question I’ve never quite been able to shake: is artificial intelligence actually an existential threat to us, or has that story become too useful for the people building it?
I’m increasingly suspicious of the way we talk about AI risk. I don’t think AI is harmless, and I certainly don’t think we should leave the industry alone to do whatever it wants. But the most dramatic version of the story gets an enormous amount of attention, while the risks already causing damage are treated almost like a footnote.
There’s a strange contradiction at the centre of the AI industry. The same labs racing to build artificial general intelligence (AI that can match people across most kinds of thinking) regularly tell governments, journalists and the public that what they’re creating could eventually threaten humanity. At the same time, they’re raising vast sums of money on the promise that these systems will become powerful enough to transform huge parts of the economy.
I find it difficult to separate those two things.
Spend enough time building and selling software and you develop a fairly good instinct for noticing when a warning also happens to be an advertisement. Saying “our technology might become powerful enough to threaten civilisation” sounds like caution, but it also says something extremely convenient to investors: our technology is going to become unbelievably powerful.
That’s quite a marketing message.
The same dynamic appears when the conversation turns to regulation. If AI is dangerous enough that only heavily licensed, carefully audited organisations should be allowed to develop frontier models, who benefits from that world? Probably not the small research lab or startup trying to compete with companies that already have billions in funding, teams of lawyers and entire departments dedicated to compliance.
Regulation can be necessary and still protect incumbents, and pretending otherwise strikes me as remarkably naive. The companies calling most loudly for sophisticated regulatory regimes are often the companies best equipped to survive them.
There’s another useful feature to the AGI story too: nobody has to say exactly when any of this is supposed to happen. AGI can remain two, five or twenty years away while every disappointing product release is dismissed as another step towards something much bigger. The destination stays just far enough over the horizon that it can keep supporting the valuation without ever having to arrive.
I don’t think this means everyone talking about existential risk is lying. That explanation is too simple, and there are plenty of researchers who have worried about powerful AI systems for far longer than there has been serious money in them. Nick Bostrom was writing about superintelligence years before ChatGPT turned AI into dinner-table conversation, and academics and safety researchers have built careers around these questions without an IPO waiting at the end.
Some of them genuinely believe the risk is severe. I have no reason to doubt that.
What I do doubt is that the prominence of a risk tells us anything about its likelihood. Industries don’t discuss every problem equally. They talk about the ones that are interesting, politically useful, commercially convenient or easy to package into a story people will repeat.
An AI that becomes more intelligent than humanity and decides to seize control of civilisation is an incredible story. Data centres consuming enormous amounts of electricity, graduates struggling to find junior work and customer service teams quietly being replaced by software are not. One gets documentaries and conference panels. The other gets buried in quarterly reports.
The gap would matter less if the dramatic story were the likely one, but I don’t see much evidence that today’s models are anywhere close to becoming independent actors with their own persistent goals.
I use these systems constantly. They’re enormously useful and sometimes astonishing. They’re also brittle in ways that become obvious the moment you ask them to do something complicated for long enough.
Give a modern model a bounded programming task and it can be excellent. Give it a genuinely open-ended problem involving ambiguity, changing information and a long chain of decisions, and things become much less magical. It forgets context, misunderstands instructions, confidently takes the wrong branch and occasionally spends a surprising amount of effort solving a problem nobody asked it to solve.
Anyone who has spent much time with coding agents has probably experienced this. One minute you’re watching the machine produce several hundred lines of useful code in seconds. Ten minutes later you’re explaining for the fourth time that it shouldn’t rewrite an unrelated part of the application.
AI will almost certainly become more capable. My point is that we should distinguish the systems we actually have from the systems people imagine when discussing existential risk.
Today’s models don’t wake up in the morning with plans, or sit around between prompts thinking about what they’d like to accomplish next. Nor do they independently accumulate resources, build organisations or conduct years of research because they’ve decided they want more power. We can build software around models that gives them memory, tools and autonomy, but that’s an engineering system humans have assembled, not a spontaneous property appearing inside the model.
The recursive self-improvement story requires several more leaps. An AI would need to become exceptionally good at AI research, make meaningful improvements to its own design, successfully deploy those improvements and repeat the process without the chain breaking down. It would need access to computing resources, infrastructure and the physical systems required to turn those improvements into something real.
Perhaps we’ll build something capable of all that eventually. But “perhaps” is doing an enormous amount of work.
Meanwhile, we barely need to speculate about the damage AI can cause today because we’re already watching it happen.
Synthetic text, images, audio and video are making an already polluted information environment considerably worse. Producing plausible nonsense at scale has become incredibly cheap, and verifying it remains expensive. The economics favour the person creating the rubbish.
Phishing and social engineering have changed for similar reasons. Writing a convincing scam email used to require at least a reasonable command of the language and enough effort to tailor the message to its target. A language model can now produce thousands of variations, rewrite them for different audiences and remove many of the obvious linguistic clues people once relied on.
None of this requires consciousness, AGI or even particularly good AI. It just requires automation that is cheap and convincing enough.
The same thing is beginning to happen to work. I’m much less interested in the question of whether an AI will some day replace every software engineer than I am in what happens when one engineer with good AI tools can comfortably do work that previously required three people. The second scenario is dramatically more plausible, and companies don’t need to believe in AGI before acting on it.
That change won’t arrive as one enormous moment where machines suddenly take everyone’s jobs. It’ll happen quietly. A team of twelve becomes nine. The next junior role isn’t opened. A support department handles twice as many customers without hiring anyone. A freelance writer discovers that half the low-end work has disappeared because “good enough” text now costs almost nothing.
These are the boring economic shifts that happen slowly enough to avoid a dramatic headline while still changing people’s lives.
There’s another risk that deserves far more attention than it gets: power.
A small number of companies are building the infrastructure through which an increasing amount of human knowledge and work flows. They’re training the models, operating the compute, controlling the APIs and deciding what those systems will and won’t do. As AI gets embedded into search engines, operating systems, development tools and office software, that concentration becomes more important.
I’m far more uncomfortable with that than I am with a hypothetical machine secretly plotting humanity’s extinction.
Software has always mediated what people can see and do, but AI pushes that mediation further. When a search engine gives you ten links, you can inspect the sources and decide where to go. When an assistant gives you one synthesised answer, an enormous amount of filtering has already happened before you see a single word.
It matters who controls that filtering, which sources models favour, what they refuse to discuss and whether competitors can afford to build alternatives. Those are governance problems happening now rather than thought experiments about systems that don’t exist yet.
None of this means I think long-term AI safety research is pointless. Preparing for low-probability but extremely damaging outcomes is perfectly reasonable, especially when we’re dealing with technology whose capabilities are advancing quickly. Software engineers think about failure modes before they happen all the time. That’s part of the job.
What bothers me is the proportion of attention.
The public discussion has somehow jumped from “language models can generate remarkably convincing text” to “we need to prepare for machines exterminating humanity”, skipping over an enormous collection of much more ordinary problems in between. Those ordinary problems aren’t as exciting, but they’re the ones we can actually measure, argue about and do something about today.
I also don’t put much weight on arguments from authority here. Some extremely respected researchers are deeply concerned about catastrophic AI risk. Other equally accomplished researchers think those fears are exaggerated. Being brilliant at machine learning doesn’t give anyone access to knowledge about a future technology that hasn’t been invented yet.
Expert concern is a reason to investigate a problem. It isn’t evidence that the predicted outcome will occur.
That’s roughly where I’ve landed after a few years of using these models almost every day. AI is powerful technology that deserves serious scrutiny, sensible regulation and a great deal more attention than the usual Silicon Valley “move fast and apologise later” approach. I just don’t think an imaginary superintelligence should dominate that conversation.
The apocalypse narrative is too convenient. It makes AI companies sound all-powerful, turns executives into people supposedly carrying the future of humanity on their shoulders and gives governments a reason to build regulatory structures that the largest companies are uniquely capable of navigating. Maybe some version of the nightmare eventually turns out to be correct, but extraordinary future scenarios shouldn’t distract us from mundane present ones.
I’d rather spend our attention on the AI problems we can actually see.
People are losing work, scammers are getting better tools, and synthetic content is making it harder to trust what we see online. Meanwhile, infrastructure and influence are concentrating inside a small collection of companies. None of that requires a singularity or an evil machine plotting in a data centre.
It only requires us to keep deploying the technology we already have.