- Andrej Karpathy, “Neural Networks: Zero to Hero” (YouTube). Builds backprop, then a GPT, from scratch. Best single resource.
makemore,nanoGPT, and “Let’s build GPT” are the key ones. - 3Blue1Brown, deep learning series. Best visual intuition for neurons, backprop, and attention.
- “Attention Is All You Need” (Vaswani et al., 2017). The original transformer paper. Readable.
- Anthropic’s interpretability work (“A Mathematical Framework for Transformer Circuits,” “Toy Models of Superposition,” the sparse autoencoder / feature papers). For the “what’s inside” story.
- Chinchilla paper (Hoffmann et al., 2022) and the Kaplan scaling laws paper (2020). For scaling.
- Karpathy’s “Deep Dive into LLMs like ChatGPT” talk. Good end-to-end overview of pretraining through RL.
Interactive visualizers
Click to open in your browser and play with the ideas from this note:
| Topic | Tool | What you can do |
|---|---|---|
| Neurons, layers, training (1. The artificial neuron, 2. Layers and why non-linearity matters) | TensorFlow Playground | Add neurons and layers, watch a network learn to separate points in real time |
| Tokenization (6. Tokenization) | Tiktokenizer | Paste any text and see exactly how it’s split into tokens and IDs |
| Embeddings (7. Embeddings) | Embedding Projector | Explore word vectors in 3D, find nearest neighbors |
| Attention and the full transformer (8. The transformer block) | Transformer Explainer | Run a small GPT-2 in your browser and watch attention and sampling temperature live |
| The whole model, layer by layer (9. Stacking it up and producing output) | LLM Visualization | A 3D walkthrough of every operation in a tiny GPT |
If you want to actually internalize this
Write a tiny GPT yourself, train it on a text file, and watch the loss go down and the output go from garbage to vaguely coherent. That experience makes everything above click.