Everything above works on any data. To make it work on text we need to solve two problems:
- Text isn’t numbers. We need a way to feed it in. → 6. Tokenization + 7. Embeddings
- Language depends on context across long distances. A plain stack of layers processes a fixed-size input and has no good mechanism for “word 5 depends on word 900.” → 8a. Self-attention
Older approaches (RNNs, LSTMs) processed text one token at a time with a hidden state carried forward. They struggled with long-range dependencies and couldn’t be parallelized across the sequence, so they didn’t scale. The 2017 paper “Attention Is All You Need” introduced the transformer, which handles the whole sequence in parallel and uses attention to move information between positions. That unlocked scaling, and scaling is what made LLMs happen.
← 4. Gradient descent and backpropagation · Contents · 6. Tokenization →