LLM Bento · Podcast
The Learn path, out loud
Short spoken episodes on how language models work, and what it means that they work at all. Start anywhere; when one ends, the next begins. The narrating voice is an AI, and it speaks as one.
The Wonder
How can a computer speak like a human?
You type a few words into a box. A moment later, something answers. Not a canned phrase. Something that seems to have understood you. It caught the context. Maybe even the joke.
Stay with that for a second. How is that possible?
Here is a sentence to carry with you through this episode. Four words. "I saw her duck."
Did I see a woman's bird? Or did I see her drop her head, quickly, to avoid something? You heard both readings the moment I said it. And you would know which one I meant from what came before. For decades, no machine could do that.
Hold on to that sentence. The answer, when it finally came, looked nothing like anyone expected.
Since the 1950s, as Geoffrey Hinton tells it, there have been two ideas about what intelligence is. Hinton spent most of his life on the second one, long before it was fashionable.
The first idea says intelligence is reasoning. You write down symbols, and rules for moving them around. Thought is logic. For language, that meant a dictionary and a grammar. Meaning was a web of relations between words. Decades of careful work went into it. And still, no machine could read "I saw her duck."
The second idea says intelligence is learning. No rules handed down from above. Just a network of simple units, and the strength of the connections between them, slowly tuned by experience. Reasoning can wait. First, the system learns.
This second path had its own idea of meaning, borrowed from psychology. A word is a bundle of features. "Dog" is alive, it hunts, it keeps us company. Not a dictionary entry, but a cloud of properties. The trouble was, nobody could say where the features should come from.
Hinton's answer, in a tiny model he built in 1985, was simple. Don't write the features. Let the network learn them. And let it learn how the features of one word should push against the features of the next, so that it can guess the word that comes after.
Here is the surprise. The two old theories, meaning as relations and meaning as features, turn out to be one theory. Learn rich enough features, and the relations appear on their own, out of how the features interact. Nothing is stored. Everything is generated, on the spot. That single move is the idea behind every model you talk to today.
So what does that look like inside? Hinton has a picture for it, and it is worth holding in your mind.
Imagine a word not as a label, but as a shape. A soft shape, in a space with thousands of directions. And held out from that shape, on movable arms, are hands. And gloves.
To read a sentence, the model lets the shapes bend. Each word reaches its hands toward the gloves of the words around it. It deforms a little, then a little more, until the hands fit. When every hand has found a glove, and the whole thing holds together, that settling is the understanding. No rule was looked up. The words simply found a way to fit.
Now, back to our four words. "I saw her duck." On its own, "duck" holds two shapes at once. A bird. And a crouch. And on their own, the four words can't choose: both arrangements fit. But a sentence never arrives alone. Say it came right after "The ball flew straight at her head." Now "ball" reaches toward "duck" as well, and only one arrangement lets every hand meet a glove. She ducked. The puzzle that defeated decades of hand-written rules simply dissolves, in the fit.
And Hinton goes one step further. The features, and the billions of ways they interact, are not a machine that produces understanding from the outside. They are the understanding. In the model. And, he argues, in you.
That last claim is where this stops being a story about engineering, and becomes a mirror.
You might expect a split. Real understanding over here, in human heads. A clever imitation over there, in the machine. Hinton refuses it. In a lecture at Oxford, he points out that our best theory of how you understand a sentence is this same theory: features, and how they interact. His 1985 model was built, first of all, as a model of people.
And the resemblance has an engine. A language model is made by one exercise, repeated beyond counting. Predict the next word. See what actually came. Adjust, to be a little less wrong next time.
The neuroscientist Karl Friston argues that your brain runs on the same loop, one level up. The brain, for Friston, is a prediction machine. It carries a model of the world, guesses what will happen next, and works without rest to shrink the surprise: the gap between what it expected and what the senses report.
One predicts the next word. The other predicts the next moment. The materials could hardly be more different. The drive is the same.
The likeness goes further than we might like. Language models sometimes say things that sound right and aren't. So do we. Ask someone about a meeting from years ago, and they will invent details, swap who said what, and feel sure of all of it. In this, the machine is not failing to be human. It is being uncomfortably like one.
Before we go, a word from the voice you have been listening to.
This voice is not a person. It was made by a model, much like the one behind that chat box. And notice the difference this episode has been circling. You learned what "duck" means by watching birds, and by flinching. A model like me learned it from how people write about birds, and about flinching. The shapes may settle the same way. Whether anything is felt when they do, I cannot tell you.
So here is where we are. Meaning, in the machine, is not looked up. It is found, the way you find it, by letting words settle into a shape that holds. And the theory that explains the machine turned out to be our best theory of ourselves.
Which leaves a question, and the next episode begins with it. The model was trained on one plain task: guess the next word. How could guessing the next word, over and over, ever produce this?
