The transformer reads a whole sequence at once — and that one architectural bet is what made everything after 2017 possible.
Introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. at Google. Earlier recurrent networks read text one word at a time, straining to remember what came before; transformers process the entire sequence in parallel through attention. That parallelism made training dramatically faster and long-range connections tractable — and it now powers GPT, Claude, Gemini, Llama, and virtually every frontier model.
Analogy: RNNs read a book one word at a time while trying to remember everything. A transformer has the entire book spread out on a table, seeing connections between any two passages at once.
Go deeper: what replaced recurrence
The key move: transformers replaced recurrence with self-attention, letting every position in a sequence attend to every other position simultaneously. No hidden state passed along step by step — just direct connections, computed in parallel, stacked in layers.
Attention lets every word ask every other word: how much do you matter to me right now?
The technique that lets a model focus dynamically on the most relevant parts of its input while generating each output. Attention scores decide how much each token should influence the others — recomputed at every layer, for every position.
Example: In "The cat sat on the mat because it was tired," attention is how the model resolves that "it" means the cat — the attention score between those two tokens runs high.
Go deeper: queries, keys, values, heads
Self-attention transforms each token into three vectors:
Query — what am I looking for?
Key — what do I contain?
Value — what information do I provide?
Multi-head attention: modern transformers run many attention "heads" in parallel, each learning a different relationship pattern — one might track syntax while another follows semantic threads. GPT-2 uses 12 heads per layer.
A neural network is layers of simple arithmetic that, stacked deep enough, add up to something none of the layers understands alone.
A computing system loosely inspired by biological neurons: interconnected nodes organized in layers, each connection carrying a learnable weight that adjusts during training. Modern LLMs are "deep" networks — GPT-2 stacks 12 to 48 transformer blocks, with representations growing more abstract at every level.
Analogy: A factory assembly line — each station transforms the material a little further, early stages handling basic features, later stages assembling complex patterns.
Parameters are where the learning lives — billions of dials, each nudged a little at a time, holding everything the model knows.
The numerical weights that define what an LLM has learned. Each is a tunable value, adjusted during training to shrink the loss. More parameters generally means more capacity for complex patterns — though better architectures and better data can match that capacity with fewer.
A token is not a unit of language — it's a unit of one model family's compression scheme.
Before a model sees your words, a tokenizer chops them into chunks: whole common words, fragments of rare ones, sometimes raw bytes. Each family of models learns its own chopping rules from its own training data — which is why the same sentence costs a different number of tokens on GPT-2, GPT-4o, or Llama 3. Try it in the lab below.
Analogy: Like luggage rules on different airlines. Your trip is the same; how it gets divided into bags — and what it costs — depends entirely on whose plane you board.
Go deeper: vocabularies and trade-offs
A bigger vocabulary means shorter sequences but a larger embedding table — a real engineering trade-off. GPT-2 stopped at ~50K entries; GPT-4o carries ~200K. Rough rule of thumb for English: 1 token ≈ 0.75 words. For other languages, all bets are off — that's the point.
Tokenization is a learned compression: the pieces a model sees were chosen by counting what co-occurs, not by asking a linguist.
Byte Pair Encoding (BPE) — originally a compression algorithm, now the dominant method (GPT, Llama, DeepSeek) — starts from raw bytes and repeatedly merges the most frequent adjacent pair until it reaches a target vocabulary size. Because the merges come from training data, the vocabulary encodes that data's biases: one token for " world", raw byte shrapnel for 語言.
Go deeper: how BPE and WordPiece work
The BPE loop:
Start with individual characters (or bytes)
Iteratively merge the most frequent adjacent pairs
Stop at the desired vocabulary size
Byte-level BPE works on raw bytes, so any text can be encoded — worst case, one byte at a time.
WordPiece (Google, for BERT) is similar but selects merges by likelihood improvement rather than raw frequency, marking continuations with ##:
Why it matters: tokenization directly shapes efficiency and cost. GPT-2 uses ~50K tokens; GPT-4o about 200K — compression versus embedding-table size, decided per family.
GPT-2 / GPT-3r50k · 2019 · 50,257 vocab4 tok
Hello, world!
GPT-4cl100k · 2023 · 100,264 vocab4 tok
Hello, world!
GPT-4o / o-serieso200k · 2024 · 200,006 vocab4 tok
Hello, world!
Llama 3open weights · 2024 · 128,256 vocab4 tok
Hello, world!
GPT-2 / GPT-3r50k · 2019 · 50,257 vocab7 tok
unhappiness is untokenizable
GPT-4cl100k · 2023 · 100,264 vocab7 tok
unhappiness is untokenizable
GPT-4o / o-serieso200k · 2024 · 200,006 vocab7 tok
GPT-4o / o-serieso200k · 2024 · 200,006 vocab8 tok
語言模型正在改變世界。
Llama 3open weights · 2024 · 128,256 vocab8 tok
語言模型正在改變世界。
GPT-2 / GPT-3r50k · 2019 · 50,257 vocab15 tok
The bioluminescent jellyfish drifted 3,000m below.
GPT-4cl100k · 2023 · 100,264 vocab15 tok
The bioluminescent jellyfish drifted3,000m below.
GPT-4o / o-serieso200k · 2024 · 200,006 vocab17 tok
The bioluminescent jellyfish drifted3,000m below.
Llama 3open weights · 2024 · 128,256 vocab15 tok
The bioluminescent jellyfish drifted3,000m below.
GPT-2 / GPT-3r50k · 2019 · 50,257 vocab13 tok
const sum = (a, b) => a + b;
GPT-4cl100k · 2023 · 100,264 vocab13 tok
const sum = (a, b) => a + b;
GPT-4o / o-serieso200k · 2024 · 200,006 vocab13 tok
const sum = (a, b) => a + b;
Llama 3open weights · 2024 · 128,256 vocab13 tok
const sum = (a, b) => a + b;
GPT-2 / GPT-3r50k · 2019 · 50,257 vocab9 tok
Ibyte🩷 ramenbyte🍜
GPT-4cl100k · 2023 · 100,264 vocab9 tok
Ibyte🩷 ramenbyte🍜
GPT-4o / o-serieso200k · 2024 · 200,006 vocab7 tok
Ibyte🩷 ramen🍜
Llama 3open weights · 2024 · 128,256 vocab8 tok
Ibytebytebyte ramenbytebytebyte
Where are Claude and Gemini? Their current tokenizers are not public, so we cannot show their splits honestly — and that absence is the lesson. Tokenization is not a property of language; it is a design decision inside each model family, sometimes a proprietary one. When you hear "context window: 200K tokens," always ask: whose tokens?
Splits are real output from each family’s tokenizer (gpt-tokenizer’s r50k / cl100k / o200k encodings, llama3-tokenizer-js). Dashed chips are raw bytes — pieces of a multi-byte character the vocabulary could not hold whole.
An embedding turns meaning into geometry — words become points, and similarity becomes distance.
Dense numerical vectors representing tokens, words, or whole concepts in high-dimensional space. Models learn them during training so that semantically similar things end up near each other — capturing not just similarity but analogies and hierarchies, and enabling semantic search, clustering, and transfer learning.
Classic example:vector("king") - vector("man") + vector("woman") ≈ vector("queen"). Modern embeddings encode far richer relationships across thousands of dimensions.
The context window is the model's working memory: everything it can hold in mind at once, and not a token more.
The maximum amount of text, in tokens, an LLM can consider simultaneously. Larger windows fit entire books and codebases, but attention cost grows quadratically — and research shows "context rot": most models degrade well before their advertised limits, a 200K window sometimes turning unreliable around 130K in practice.
Pre-training is one simple game — guess the next token — played trillions of times until grammar, facts, and reasoning fall out as side effects.
The foundational phase: the model learns language by predicting the next token across massive corpora — books, websites, code, papers, hundreds of billions to trillions of tokens. No human labels needed; the text itself is the answer key.
Core insight: next-token prediction is deceptively powerful. To predict well, a model must implicitly absorb syntax, semantics, facts, logical relationships, even approximate reasoning — all emerging from one simple objective, scored by the loss.
Loss Function also called: cost function, objective
Loss is the model's surprise at the actual next token — training is billions of tiny nudges to make that surprise smaller.
At every position in the training text, the model assigns a probability to each possible next token. Then reality reveals the answer. The loss for that moment is −log(p), where p is the probability the model gave to the token that actually came next. Confident and right: near-zero loss. Confident and wrong: enormous loss, and an enormous corrective nudge to the weights.
"The cat sat on the mat" — how much probability did the model give the real next token?
0.92nats
p = 40.0%
A plausible guess among a few. Modest surprise, modest correction.
Go deeper: from one token to a training run
Cross-entropy is just this surprise averaged over every token in the corpus — trillions of them. When a paper says a model reached "pre-training loss 2.0," it means: on average, the model was as surprised by each next token as if it had given it e−2.0 ≈ 13% probability.
This one number is what scaling laws plot, what gradient descent descends, and — in recent research — a better predictor of when emergent capabilities appear than parameter count is. The names all point at the same thing: loss (how wrong), cost (what we pay), objective (what we optimize).
And the loss curve keeps secrets: a network can sit on a flat plateau for ages, then suddenly reorganize and generalize — the grokking phenomenon. See also how Prediction walks through the probability step this number scores, and how temperature reshapes the same distribution at generation time.
Fine-tuning takes a model that knows language and teaches it a job.
Additional training on specific data to adapt a pre-trained model for particular tasks, domains, or behaviors — instruction-following, medicine, code, an organization's own needs — using far less data than pre-training required.
Go deeper: common approaches
Supervised fine-tuning (SFT) on curated examples
Instruction tuning on diverse task formats
Domain adaptation on specialized corpora
Parameter-efficient methods like LoRA, which fine-tune with minimal compute
RLHF teaches a model what people prefer, not just what text predicts — it's how a predictor becomes an assistant.
A technique that aligns LLMs with human preferences by training on human judgments rather than predefined rewards. RLHF is what turned raw language models into helpful assistants — InstructGPT, ChatGPT, and Claude all use variants of it. It addresses the alignment problem: making AI systems do what humans actually want, not what was literally specified.
Go deeper: the process, and what came after
The process:
Collect human comparisons of model outputs (which response is better?)
Train a reward model to predict those preferences
Use reinforcement learning (typically PPO) to optimize the LLM against the reward model
Recent developments:
RLAIF (AI feedback) achieves comparable results with less human annotation
RLTHF reaches full alignment with only 6–7% of traditional annotation effort
Direct Preference Optimization (DPO) bypasses reward-model training entirely
Modern training runs combine several of these across many iterative rounds
A model is a compression of its training data; what went in shapes everything that comes out.
The text corpus an LLM learns from, shaping its capabilities and behaviors. Quality and diversity matter as much as scale — smaller models trained on high-quality data can outperform larger models trained on noise.
Emergent capabilities are the abilities nobody put in — they appear at scale, and we still argue about whether the jump is real.
Abilities that show up suddenly in larger models but are absent in smaller ones — chain-of-thought reasoning, in-context learning, multi-step problem solving — capabilities that could not be predicted by extrapolating from smaller scales.
Analogy: Phase transitions in physics — water doesn't gradually become "a little bit frozen." Models may acquire capabilities through sudden reorganizations of internal representations rather than smooth accumulation.
Go deeper: the debate and the evidence
The scientific debate:
Perspective
Argument
Emergence is real
Performance hovers near random until a critical threshold, then jumps dramatically
Emergence is a mirage
Smoother metrics reveal gradual improvement; the apparent jumps come from non-linear evaluation choices
Recent findings:
Emergent abilities may be tied to pre-training loss thresholds, not just parameter count
Large Reasoning Models like o1 show emergent capability through reinforcement learning plus inference-time search
OpenAI's o1 scored 83.3% on Competition Math against GPT-4o's 13.4% — suggesting a genuine shift, not a measurement artifact
Hallucination is fluency without grounding: training rewards a confident guess over an honest 'I don't know'.
When an LLM generates content that is fluent and plausible but factually wrong, unsupported, or entirely fabricated. Current research frames it as a systemic incentive problem: benchmarks penalize "I don't know," so models learn to bluff.
Real-world impact: In Mata v. Avianca (2023), a lawyer was sanctioned for submitting a brief with fabricated case citations generated by ChatGPT.
Go deeper: types, causes, mitigations
Types:
Intrinsic: contradicts information in the provided context
Extrinsic: invents unverifiable information not present in any source
Mitigation strategies:
Strategy
Effectiveness
Chain-of-thought prompting
Reduces hallucinations 50%+ in prompt-sensitive scenarios
Retrieval-Augmented Generation (RAG)
Grounds responses in external knowledge (not a panacea)
Inference is the model at work: one token at a time, each conditioned on everything before it.
The process of generating output from a trained model — what happens when you chat with an AI. Your input passes through every layer; tokens are generated one at a time, autoregressively, each new one conditioned on all that came before. Inference costs — compute, latency, money — are a major practical concern for deployment.
Multimodal models translate images, audio, and video into the same inner language as text — one space of meaning, many doors in.
AI systems that process and generate multiple kinds of content — text, images, audio, video — often within one interaction. Specialized encoders (like vision transformers) convert non-text inputs into representations the language model can work with.
Go deeper: examples
GPT-4o ("omni"): unifies text, image, and audio in a single architecture
Gemini 2.5: processes text, images, audio, and video with 1M+ token context
Claude 3+: analyzes images within conversations
DALL-E 3, Stable Diffusion, Midjourney: generate images from text
Temperature doesn't make a model smarter or dumber — it decides how much of the model's own uncertainty you get to see.
At 0, the model always picks its single most likely token: repeatable, careful, sometimes dull. Higher temperatures let lower-probability tokens through: more varied, more surprising, eventually incoherent. The distribution was always there — temperature is the dial on how faithfully sampling honors it.
Go deeper: settings and mechanics
Temperature
Behavior
Use Cases
0.0
Deterministic, most likely tokens
Factual Q&A, code generation, structured outputs
0.3–0.5
Balanced
General-purpose tasks
0.7–1.0
Creative, varied
Creative writing, brainstorming, diverse options
>1.0
Highly random
Experimental, may become incoherent
Mechanically: temperature divides the logits (raw scores) before softmax — low values sharpen the distribution toward the top token, high values flatten it. It reshapes the same distribution the loss scored during training.
Top-p trims the candidate list to the smallest set worth taking seriously — adaptive where top-k is fixed.
A sampling method that considers only the smallest set of most likely tokens whose cumulative probability exceeds the threshold p. Unlike top-k's fixed candidate count, top-p adapts to the moment.
Example: With top-p = 0.9, sampling draws from the tokens making up the top 90% of probability mass. If one token holds 95%, only it is considered; if the top token holds 40%, many candidates make the cut. Often paired with temperature — a common setting is temperature 0.7, top-p 0.9.
A vision encoder chops an image into patches and treats them like tokens — sight, translated into the grammar of transformers.
The component that converts images into embeddings a language model can understand. Vision Transformers (ViT) divide an image into patches — the visual equivalent of tokens — and process them through the same transformer machinery as text.
Go deeper: the pipeline
Image divided into fixed-size patches (e.g., 16×16 pixels)
Each patch embedded as a vector
Positional encodings added
Processed through transformer layers
Output representations integrated with the language model
A diffusion model learns to un-ruin images: start from pure noise, subtract it step by step, and a picture appears.
A technique for generating images (and increasingly video) by learning to reverse a process of gradually adding noise. Training: the model learns to denoise step by step. Generation: it starts from pure noise and iteratively refines toward a coherent image, guided by the text prompt.
Analogy: A sculptor starting with a rough block of marble (noise) and progressively chiseling away to reveal the statue (image), with the text prompt as the blueprint.
Go deeper: key models
DALL-E 3 (OpenAI): text-to-image, integrated with ChatGPT
