The data
Trillions of tokens: filtered web crawl, books, academic papers, code, forums, Wikipedia, and increasingly synthetic data. A huge amount of effort goes into data quality: deduplication, filtering junk, balancing domains. Data curation is arguably as important as architecture.
The objective
Just next-token prediction with cross-entropy (the formula from 3. Loss and how wrong we are). That’s it. Take a chunk of text (say 8k tokens), run it through, compute the average loss over all positions, backprop, update.
The training loop
for step in range(hundreds_of_thousands):
batch = sample many sequences from dataset
logits = model(batch.inputs)
loss = cross_entropy(logits, batch.targets) # targets = inputs shifted by 1
loss.backward() # backprop
clip gradients # stability
optimizer.step() # AdamW update
scheduler.step() # adjust learning rate
Why this teaches anything beyond autocomplete
To predict the next token well across everything humans write, the model gets forced to learn whatever internal structure reduces loss:
- Grammar and syntax (obviously helps)
- Facts about the world (“The capital of France is ___”)
- Code semantics (to predict the next line of code, track variables and types)
- Arithmetic patterns, logic, causal structure of stories
- Models of who is speaking and what they likely know or want
Nobody hard-codes any of it. It’s all pressure from the loss function.
The single most important idea
A simple objective, at scale, forces rich internal representations.
Scaling laws
Empirically, loss falls as a smooth power law in three things: parameters (), training tokens (), and compute (). This predictability is why labs can plan billion-dollar training runs.
The Chinchilla paper (2022) found that for a fixed compute budget, models were being trained on too little data, with roughly 20 tokens per parameter being compute-optimal:
In practice, modern models get trained on far more than that (an overtrained smaller model is cheaper to run, and inference cost dominates over a model’s lifetime).
Training compute is roughly:
(FLOPs = floating-point operations, i.e. individual multiplies and adds.) For a 70B model on 15T tokens:
That’s thousands of GPUs running for weeks to months.
How it physically runs
No single GPU holds the model plus its optimizer state. So training is split across thousands of chips:
| Technique | What’s split up |
|---|---|
| Data parallelism | Each GPU group gets different batches; gradients get averaged |
| Tensor parallelism | Individual matrix multiplies are split across GPUs |
| Pipeline parallelism | Different layers live on different GPUs |
| Sharding (ZeRO / FSDP) | Optimizer state and weights are partitioned across devices |
| Mixed precision | Compute in bf16/fp8 for speed, keep master weights in fp32 for accuracy |
A lot of the actual difficulty in frontier training is plain infrastructure: hardware fails constantly at this scale, loss spikes happen, and runs need checkpointing and recovery.
Emergence
Some abilities (multi-step arithmetic, in-context learning, instruction following to a degree) seem to appear fairly abruptly as models scale. There’s real debate over how much of this is true phase-change behavior versus an artifact of how we measure it. Be skeptical of anyone who claims to fully understand it.
← 9. Stacking it up and producing output · Contents · 11. Post-training →