I listened to Sequoia Capital’s recent interview with Richard Sutton and Khurram Javed. If you haven’t heard it yet I highly recommend it.
https://podcasts.apple.com/us/podcast/training-data/id1750736528?i=1000784018357
As I listened, I found myself agreeing with Sutton on the main point and getting stuck (and a bit afraid) on what happens when he is proven right.
Sutton is one of the pioneers of reinforcement learning and the author of The Bitter Lesson. His argument in the interview is that we’ve built AI in a pretty strange way. We spend enormous amounts of compute training a model, deploy it into the real world, and then mostly stop letting it learn.
Sure, it can use context. We can bolt on memory, retrieval, tools, and all sorts of other things. But the weights containing what the model has actually learned generally aren’t changing because of what happens to it after deployment.
Sutton puts it plainly:
All learning is continual.
There’s a funny irony to what happens next.
The hosts of the episode, Sonya Huang and Alfred Lin keep pushing him back toward the way AI works today. What about synthetic data? What about supervised learning? Humans inherit knowledge from other humans. Nobody rediscovers calculus or rocket propulsion by wandering around and experimenting with reality.
These are reasonable objections but after a while I noticed that Sutton kept introducing a different way of thinking about intelligence and the interviewers kept trying to fit it into the framework they already had.
They had been trained extremely well on modern AI.
Sutton was asking them to update their weights.
They weren’t doing a great job of it.
Maybe I am being a little unfair, since their job is to challenge him, and the pushback made the interview better. Still, for a conversation about intelligent systems becoming very knowledgeable and then losing their ability to meaningfully update themselves, it was pretty funny.
The annoying part is that I think Sutton wins the argument. So I would have loved to see the conversation move past their sticking points and onto meatier topics with real complex problems to solve.
We train the model and then freeze it
Imagine an extraordinarily educated 25-year-old.
They’ve read nearly every book ever written. They know mathematics, history, physics, medicine, software engineering, philosophy. They speak dozens of languages.
Then something happens and they can no longer learn.
They can hold information in their head while you’re talking to them. They can keep notebooks. They can search those notebooks later. Give them access to a giant library and they can look up almost anything.
But their actual understanding of the world can never change again.
They can’t develop a new intuition. They can’t discover that something they believed for years was wrong and rebuild their understanding around it.
That would be a bizarre form of intelligence.
It’s also not that far from what we’re building.
During training, a model develops an enormous amount of conceptual structure in its weights. Then we deploy it. Now it’s interacting with millions of people and encountering things that weren’t in its training data, but those interactions generally don’t change the weights.
I’ve had a hard time coming up with a good first-principles reason why we’d want learning to stop there.
There are plenty of practical reasons. Stability. Cost. Safety. Catastrophic forgetting. Being able to test the thing you’re putting into production.
Those are very good reasons.
They don’t make Sutton wrong about intelligence.
The internet eventually runs out
Sutton and Javed call this the Big World Hypothesis.
We’ve put a ridiculous amount of human knowledge online, but the internet is still a finite record of things that already happened. It is also heavily filtered. Someone had to think something was worth writing down, measuring, photographing, filming, uploading, or labeling.
Even if we continue with offline datasets (ala Anthropic with rare books). The world keeps producing situations that aren’t in that record.
Synthetic data is one proposed answer. When useful human-generated training data becomes scarce, models and simulations can generate more.
Sutton is skeptical of that approach, and Javed gives an example that makes the problem much clearer.
Say you’re training a drone in simulation. Engineers decide what goes into the simulator. Eventually the drone flies in the real world and something happens that the simulator didn’t account for. Maybe the physics were slightly wrong. Maybe the environment behaved in an unexpected way.
The engineers study what happened and update the simulation.
But the drone is the thing that encountered the error.
Why does a human need to take its experience, figure out what mattered, encode that back into a training environment, train another model, and send it out again?
Why can’t the drone learn?
That’s the version of Sutton’s argument I find hard to argue with.
If an AI predicts something, acts, sees what actually happens, and realizes its model was wrong, I want it to be able to learn from that.
And then I hit the part I can’t reconcile.
Six months later, is it still the model we aligned?
Today we roughly train a model, do whatever alignment and safety work we’re going to do, evaluate it, and deploy it.
I am massively simplifying this, obviously. But there’s a useful property buried in that process: the model we test isn’t normally rewriting its own weights every time somebody uses it.
Give the model the kind of learning algorithms Sutton is describing and that changes.
You align it. You deploy it. It spends six months interacting with the world and learning from those interactions.
Hopefully it is now better.
It should also be different.
Otherwise, what was the point?
Javed talks about agents developing new abstractions, which makes this more interesting than an AI learning a new fact.
Humans do this constantly. You can spend years interpreting the world one way, encounter enough contradictory evidence, and eventually realize the framework itself was wrong.
People do it with science. Religion. Politics. Parenting. Relationships. Pretty much everything.
Sometimes learning means changing your mind about a fact. Sometimes it changes how you organize a bunch of facts you already knew.
We should probably want an intelligent machine to do that.
I just don’t understand why we’d expect that ability to conveniently stop when it reaches the parts of the model we aligned.
Say we manage to encode something like “protect human life.”
I don’t know where that lives in a future AI. Maybe it’s a goal. Maybe it’s part of a reward function. Maybe it’s represented across a huge number of weights. Maybe the whole framing will sound primitive twenty years from now.
Whatever it is, can the system learn its way out of it?
If yes, alignment has become a moving target.
If no, we’ve already decided that some things are more important than a learning process in the model.
That seems like a pretty important design decision.
I didn’t care about the twitching baby
Sutton talks about babies learning motor control.
A baby isn’t handed a training set explaining which neurons to fire to move its thumb. It fires neurons, its body does something, and over time it figures out the relationship between action and result.
When I first heard this I didn’t find it particularly convincing.
Who cares if an AI can learn to twitch a muscle?
I was taking the example too literally.
The useful part is that nobody had to know beforehand which experiences mattered enough to put in a dataset.
Think about a master mechanic who hears a noise in an engine and immediately knows something isn’t right. Maybe he can’t even fully explain why yet. He’s heard thousands of engines, made thousands of diagnoses, been wrong, taken things apart, fixed them, driven the cars afterward. Somewhere in all of that experience he learned distinctions that aren’t in a service manual.
That’s the kind of learning Sutton is talking about.
A lot of human expertise is like this. We develop concepts because reality repeatedly gives us feedback, including feedback nobody thought to label beforehand.
Most AI training data has already passed through human judgment before the model ever sees it.
Sutton wants the model to encounter the thing itself.
Okay. I’m with him there.
But now let’s go back to the baby.
The baby didn’t start from scratch
Babies subscribe to what Sutton would call the learning process, but they come with a hell of a lot preinstalled.
They have eyes that work a particular way. A human body. Reflexes. Pain. Pleasure. Drives. Attention. Social tendencies. Learning biases. A brain architecture they didn’t choose.
Evolution spent billions of years producing the thing that is now doing the learning.
I originally thought epigenetics might make this point even stronger. There is evidence for transgenerational epigenetic effects, particularly in other animals, but I don’t think the evidence supports saying humans inherit learned cognitive knowledge that way.
It doesn’t matter much for the argument.
Genetics is enough.
A newborn isn’t a blank slate.
Sutton doesn’t claim otherwise, which is important. He says during the interview that prior knowledge and the learning process can coexist. His problem isn’t that we give AI prior knowledge. It’s that we give it prior knowledge and then eventually stop the underlying learning process.
I think that’s right.
But humans inherit something else along with all those priors.
We inherit limits.
Maybe the limits matter too
I can’t copy myself.
I can’t spin up 10,000 Eduardos tonight, have each one spend a year learning something different, and merge all of them back into myself tomorrow morning.
I can’t add more neurons when I run out of capacity.
I can’t decide that some parts of my brain should update 100 times faster.
I can’t transfer the exact state of my brain to someone else.
I need sleep. I need food. Pain affects what I’m willing to try. My brain fits inside my skull.
And I get, optimistically, something around a century to work with.
An AI doesn’t need to share those limits.
Sutton and Javed talk about trillion-parameter minds and copying digital agents that then accumulate different experiences.
That is where the baby analogy gets weird.
Evolution didn’t only give us a starting architecture for learning. It also gave us a physical envelope we cannot escape.
A machine gets whatever envelope we build.
We spend a lot of time treating AI constraints as things we need to eliminate. More context. More memory. Longer-running agents. More compute. Faster inference. Continual learning.
Usually removing those constraints makes the system more useful.
I’m less sure we should assume every constraint is bad.
If an AI can keep learning for years, copy itself, combine experience across copies, expand its capacity, and operate much faster than a human, then “it learns like a baby” stops being a particularly comforting comparison.
It doesn’t learn like a baby.
It learns like whatever we built.
This is where I’m stuck
I agree with Sutton that an intelligent system should be able to learn from experience after deployment.
I also don’t see how you give a system that ability while assuming the values you aligned beforehand will remain untouched.
Maybe the answer is architectural. Some parts of the system remain plastic and others don’t.
But then we have to decide what goes in the part that doesn’t move.
And that gets uncomfortable fast.
Imagine we somehow succeed at making a value permanently immutable. The AI can revise its physics, economics, mathematics, strategies, and understanding of the world, but this particular value can never change.
What happens if the system eventually understands something about morality that we don’t?
We could be freezing today’s human judgment into a system that eventually knows far more than we do.
On the other hand, if its foundational values can update like everything else, then I don’t know what alignment means over a long enough period of time.
Maybe there is a way to preserve values while allowing the representations around them to change. Maybe the goal itself can remain stable while the system gets much better at understanding what that goal means. Maybe continual alignment becomes part of continual learning.
I don’t know.
But I think that’s the question Sutton’s argument forces us to answer.
We are eventually going to want AI systems that learn from what happens to them. Sutton convinced me of that.
Before we let them, I want to know which parts of them are allowed to learn.
And which aren’t.