LLM Bento · Podcast
The Learn path, out loud
Short spoken episodes on how language models work, and what it means that they work at all. Start anywhere; when one ends, the next begins. The narrating voice is an AI, and it speaks as one.
The Core Insight
Next Token Prediction
Last time, we ended on a question. A model is trained on one plain task: guess the next word. How could guessing the next word, over and over, ever produce something that seems to understand you?
Let's start with the task itself. Finish this sentence. "The cat sat on the..."
You already did. Mat, maybe. Or floor. Or couch. You did not think elephant. You did not think tomorrow. You have heard enough sentences in your life that the next word arrived before you asked for it.
That is the whole job of a language model. This episode is about why that job turns out to be so much bigger than it sounds.
First, a surprise. The model never sees words. It does not even see letters.
Before anything else happens, the text is cut into pieces called tokens. A common word is usually one piece. A rare word gets broken into two or three. Each piece is given a number, and each number becomes a point in that space of thousands of directions we met last time, the soft shape with hands and gloves.
Where the cuts fall depends on the model. Each family of models learns its own way of cutting. And language matters. The same thought, said in Chinese, often takes more pieces than it does in English. It costs more to say.
Here is why this is worth pausing on. We met the world through a tongue, through hands, through falling down. The model meets the world as a stream of numbered pieces. Everything it will ever know arrives that way. Hold on to that. We will come back to it.
So, given the pieces so far, what does the model do? It gives every possible next piece a likelihood. After "The weather is", maybe "nice" is likely, "cold" a little less, "hot" less again. It picks one. It adds it to the text. Then it does the same thing again, with one more piece to look at.
Every reply you have ever read from a chat model was built this way. One piece at a time.
Training is the same loop, turned into a teacher. Hide the next piece. Let the model guess. Show it what actually came. Nudge every connection inside it, very slightly, toward a better guess. Then do it again, across more text than a person could read in thousands of lifetimes.
It sounds too simple. It sounds like the autocomplete on your phone. And with a small model, that is roughly what you get. Sentences that start well and wander off.
But make the model bigger, give it more text, and something changes. Abilities show up that nobody trained. It starts to write working code. It translates. It follows a chain of reasoning across several steps.
Why would guessing do that? Ilya Sutskever, one of the people who built these models, liked to explain it with a detective novel.
Imagine the last chapter. The detective gathers everyone in one room and says: the person who did this is... Now guess the next word. To get that name right, you would need the plot. The clues. Who lied, and why. The cheapest way to predict the end of a mystery is to understand the mystery.
That is the pressure inside training. To predict text well, a model is pushed to build some working picture of whatever produced the text: the facts, the reasoning, the people. Researchers call the abilities that appear this way emergent. It is not proof that the model understands the way you do. But it explains why a plain task does not stay plain. How it happens, and what exactly forms inside, is still partly a mystery. Later in this series, it gets an episode of its own.
One more invention made all of this work at scale. It is called attention, and it arrived in 2017.
Older models read one word after another, and by the end of a paragraph, the beginning had faded. Attention lets every piece look at every other piece, directly, and weigh how much it matters right now.
Try it. "The cat, which had been sleeping in the warm sun all afternoon, finally..." What comes next? Woke. Stretched. To guess that, you looked past "afternoon" and past "finally", all the way back to "cat" and "sleeping". That reaching back is attention.
And you have seen it before. In the last episode, "saw" and "her" reached toward "duck" and bent it into place. Attention is how the hands find the gloves.
There is a limit to how far back it can look. It is called the context window. Early models could hold a few pages. Today's can hold several novels at once. But whatever falls outside the window, for the model, does not exist.
A word about this episode itself.
The words you have been hearing were drafted with the help of a model like the one they describe. Each sentence arrived one piece at a time, each piece a guess, weighed against everything that came before it. You have been listening to a few thousand small predictions. Whether a few thousand predictions add up to meaning, or only to its shape, is the question the rest of this series keeps asking.
So here is the core insight. One task: guess the next piece. Scale it up, let every piece look at every other, and guessing becomes something that works like understanding.
Which brings us back to what we held on to. If everything the model knows arrived as numbered pieces of text, how different is what it knows from what you know? You learned what "hot" means by touching something. It learned "hot" from every sentence in which someone got burned. That difference is where the next episode begins.
