Nate Soares, the AI researcher who stopped saving for a future he doubts

Nate Soares

Photo: Not credited by the source (published in the If Anyone Builds It, Everyone Dies media kit) / CC0, via Wikimedia Commons

Born	1989
Occupation	Author

Nate Soares still has the retirement account he opened at Google. He has not put money into it for about ten years. The reason is not carelessness: the president of the Machine Intelligence Research Institute believes that superhuman artificial intelligence will probably kill everyone before he gets old enough to spend it, and he now says so in public with the flat calm of an engineer reading out a test result. “I am not really an AI job apocalypse guy,” he told an interviewer this autumn. “I’m an AI kills all humans guy.” The question that drives his work is the one he refuses to treat as closed: whether someone who expects the worst can still talk the people building the technology into stopping.

What makes him hard to dismiss is where he came from. Soares is not a philosopher who wandered into computing late. He studied computer science and economics at George Washington University, then wrote software for the federal government, first as a research associate at the National Institute of Standards and Technology and then as a Department of Defense contractor building tools for the National Defense University. After that came Microsoft and Google, the kind of companies he now asks governments to restrain.

In 2014, in his mid-twenties (he was born in 1989), he left Google for a small nonprofit in Berkeley, California, that had spent years asking a question most of the industry considered science fiction: how do you make a mind smarter than yours do what you meant? As a research fellow at MIRI he became lead author of the institute’s technical research agenda, written with Benya Fallenstein and later published as “Agent Foundations for Aligning Machine Intelligence with Human Interests”. In January 2015 that agenda was cited in “Research Priorities for Robust and Beneficial Artificial Intelligence”, the paper by Stuart Russell, Daniel Dewey and Max Tegmark that accompanied an open letter urging AI researchers to treat the social benefit of their systems as seriously as raw capability.

The same period produced the paper that gave his field part of its vocabulary. “Corrigibility”, written with Fallenstein, Eliezer Yudkowsky and Stuart Armstrong and presented at an AAAI workshop in early 2015, dealt with a problem that sounds simple and is not: a system capable enough to be useful may also be capable enough to resist being corrected or switched off. Soares and Yudkowsky say that paper introduced the term “AI alignment problem”, the phrase now used by the very labs whose plans they oppose.

Later in 2015 Soares succeeded Luke Muehlhauser as MIRI’s executive director, a post he held for eight years. Under him the institute worked on abstract mathematics rather than products: decision theory, logical uncertainty, the formal question of how an agent should reason about itself. He co-wrote “A Formal Approach to the Problem of Logical Non-Omniscience” with Scott Garrabrant and others, and “Cheating Death in Damascus”, a decision-theory paper with Benjamin Levinstein that appeared in The Journal of Philosophy in 2020. In 2017 he went back to Google to give a talk laying out the open problems in alignment and arguing that they were harder than the field assumed. On his blog, Minding Our Way, he also wrote Replacing Guilt, a sequence of essays on motivation that readers can now buy as a book, which made him better known in some circles for advice on how to work than for anything about machines.

The turn came in 2023. MIRI announced that it was moving its focus away from alignment research and toward warning policymakers and the public, while still supporting some technical work. Soares moved from executive director to president, and Malo Bourgon became chief executive. In 2025 Soares told The Atlantic that he was concentrating on public awareness because AI development was moving too fast for adequate safeguards to be put in place.

That shift produced If Anyone Builds It, Everyone Dies, written with Yudkowsky and published by Little, Brown in September 2025. The argument is in the title. A superintelligence built “using anything remotely like current techniques” would very likely end in human extinction, alignment research is nowhere near ready, and only international regulation can stop developers from racing toward systems the authors consider catastrophically dangerous. The book reached The New York Times Best Seller list in October 2025. A Spanish edition followed from Planeta, and Hayakawa Shobo published a Japanese translation in April 2026.

The reception exposed the weak joint in the case. Some of the sharpest objections came from people who share the authors’ worry. Will MacAskill, one of the founders of the effective altruism movement, called the book disappointing and said it relied on weak arguments built around an analogy with evolution and on an unstated assumption that AI progress will jump rather than climb. Clara Collier’s review in Asterisk ran under a title that summed up the frustration of sympathetic readers: “More Was Possible”. There is also an awkwardness that Soares rarely dwells on. MIRI spent years trying to solve alignment and then concluded it would not be solved in time, which can be read as intellectual honesty or as an admission that its own research program failed. His defense of his certainty is a chess analogy: he can be sure Magnus Carlsen would beat him without knowing which piece delivers mate. To his critics that is exactly the problem, a forecast about the ending with no account of the moves.

None of this has softened him. In October 2026, on The Most Interesting Thing in AI, the podcast hosted by Nicholas Thompson, he said he puts the odds that humanity does not have another decade at better than even, and explained the untouched 401(k). In the same conversation he insisted the outcome is not fixed. “I do everything I can to stop this eventuality. I don’t think it’s inevitable,” he said, and reached for C.S. Lewis’s essay “On Living in an Atomic Age”, which argued that if the bomb falls it should find people doing sensible, human things rather than cowering. His version: “Let it find us lobbying our governments for nuclear nonproliferation treaties.”

That is the job he has chosen. He remains MIRI’s president, spending his time on interviews, talks and briefings meant to turn a fringe research worry into an international agreement to halt the race toward superintelligence. The Google retirement account is still open. He has simply stopped betting on it.

Tags: artificial intelligence, Google, AI safety, Eliezer Yudkowsky, Machine Intelligence Research Institute, Nate Soares

Related People

Related People

Discussion