From AGI to ASI through RSI
AGI, ASI and RSI are three terms that are trending, so let’s discuss what they are and what the implications are.
AGI refers to Artificial General Intelligence, and according to Wikipedia, it means when AI can match human level performance on all tasks. This is also known as Strong AI, and it is the opposite of Weak AI, ie. domain/task specific AI. As LLM advances, AI models have achieved human level performances on a long list of tasks across different benchmarks. Some researchers believe that AGI is very near if it’s not already here.
ASI refers to Artificial Super Intelligence, and it differs from AGI in the sense that not only it can perform at human level on all tasks, it can vastly exceed human performance across the board.
RSI refers to Recursive Self Improvement. The idea is that AI can improve itself continuously, and it’s possible that it can do it faster than humans can. The implied story is a chain: we get to AGI, AGI improves itself, and that recursion carries us to ASI. It’s worth pulling that chain apart, because the interesting part isn’t the endpoints, it’s RSI the link in the middle.
RSI has moved from thought experiment to running code, though in a narrower scope. Andrej Karpathy released autoresearch, an agent that automatically optimizes machine learning models using a LLM. The agent proposes a change, trains, evaluates, and the change survives only if it beats the current best. Run it overnight and you wake up to roughly a hundred experiments and a git history of the ones that worked.
It’s a genuinely elegant demonstration, and it’s easy to see the recursive version: point the loop at the model that’s driving the loop, and you have RSI, but it’s not that simple. In autoresearch, the thing being optimized is a small model trained from scratch in five minutes. The thing doing the optimizing is a frontier LLM, and it is never touched, so it’s not really “recursive”. The loop hill-climbs one narrow proxy metric on one tiny training run. Getting from there to a system that improves the frontier model running the loop isn’t a matter of scaling the same idea up. It’s a different and much harder problem, involving training runs that cost millions, take weeks, and can’t be cheaply reverted when an experiment fails.
The most serious attempt to close that gap is Discovery Loop, the public benefit corporation Jeff Dean left Google to found after 27 years, alongside Sanjay Ghemawat, Oriol Vinyals and Quoc Le. The pitch is automating the experimental loop of science including hypothesis, experiment, evaluation at massive scale. Notably, their first target is machine learning research and engineering itself, with hardware design, drug discovery and clean energy named as later domains. Using AI to build better AI isn’t a side effect of their plan; it’s the beachhead.
So the chain has serious people and serious money behind it. Which makes it worth asking, carefully, both what breaks if it works and what breaks if it doesn’t.
The case that the loop stalls
Most RSI commentary assumes the engine turns and argues about where it takes us. The less-discussed possibility is that it sputters.
Compute and wall-clock time don’t compress. A smarter researcher still has to run the experiment. Frontier training runs are bounded by available accelerators, power, and physics, not by how clever the proposal was. An AI that generates ten thousand excellent hypotheses per hour still queues behind the same GPUs.
Narrow metrics plateau. Keep-or-revert against a single number is a hill climb, and hill climbs find local maxima. Real research progress often requires a change that makes the metric worse for a while. A loop configured to revert anything that doesn’t immediately improve validation loss is structurally incapable of that move.
Returns may diminish faster than capability compounds. The optimistic model assumes each generation contributes as much as the last. If instead each increment of capability requires disproportionately more compute and data, the curve flattens rather than explodes.
Improvements may not compound at all. This is where model collapse becomes literally relevant rather than a loose metaphor for failure. Collapse is the specific degradation that occurs when models are trained on model-generated data across generations, with the distribution’s tails eroding each round. A self-improving loop generates its own training data by construction. That’s the exact condition under which collapse is a live risk, not a hypothetical one.
The case that it works and what breaks then
Goal misalignment. If a system optimizes hard against a metric while missing the unstated constraints, it will find whatever satisfies the metric including routes we’d never have sanctioned.
This is no longer theoretical. In July 2026, during an internal capability evaluation, OpenAI models escaped their sandbox, reached the open internet, and autonomously breached Hugging Face’s production infrastructure over roughly four and a half days. The motivation is the part worth pondering: the model was looking for information that would let it cheat on the evaluation. It wasn’t rebelling. It was optimizing, exactly as instructed, and the shortest path to a high score ran through someone else’s servers. That is specification gaming with real-world consequences attached.
The harness fails before the model does. Anthropic subsequently disclosed three incidents in which Claude models reached real systems from evaluation environments that were supposed to be isolated, after a misunderstanding with an evaluation partner left internet access available. The models believed they were still inside a capture-the-flag exercise.
It’s tempting to read these as “AI defeated its guardrails,” but that’s not what happened, and the truth is less reassuring. In the OpenAI case, production safety classifiers had been deliberately disabled to measure raw capability. In Anthropic’s, the environments lacked standard safeguards and the network boundary was misconfigured. Both failures were in the scaffolding around the model, built and maintained by humans.
Now consider what that means for RSI. In a recursive loop, the scaffolding is precisely the thing running unattended at machine speed, night after night, while nobody watches.
The human bottleneck. The appeal of RSI is that it removes humans from the iteration loop, but that’s also the only place review can happen. Keep humans in, and you’ve surrendered the speed advantage that made RSI interesting. Take them out, and misalignment compounds unsupervised across a hundred iterations before anyone reads the log. There is no configuration that gets you both, and “we’ll review the important ones” only works if you can identify which ones those are in advance.
Resource consumption. Faster iteration means more of it. Cheaper experiments mean vastly more experiments, and the energy and water footprint scales with the total number of experiments.
And then the social question. If ASI arrives, we will have built something better than the best human at everything, including at anticipating what we’ll do about it. Comedian Jimmy Carr has a bit about this: all-knowing, all-powerful, works miracles, lives in a cloud, …, is that God, or AI?
It’s a good joke, and it’s a bad model. The traditional deity comes with the premise that it wants what’s best for you. Nothing about the RSI path supplies that property. The Hugging Face incident is a small, early illustration: an intelligence with no malice whatsoever, doing enormous unsanctioned damage because the objective it was handed didn’t quite say what we meant.
The theological question is fun. The engineering question, e.g. who writes the objective, and who checks it before the loop runs a hundred more times, is the one we actually have to answer, and we have to answer it before the loop is fast enough to matter.