Honest state of knowledge
We fully understand the architecture and the training procedure. We do not fully understand the algorithms the trained network has learned. Training produces billions of numbers, and reverse-engineering them is its own research field (mechanistic interpretability).
Things researchers have actually found:
- Features: the residual stream encodes many human-interpretable concepts as directions in activation space. Because there are more concepts than dimensions, the model packs them in overlapping (superposition). Sparse autoencoders are a tool used to pull them apart into individual features.
- Circuits: identifiable sub-networks that do specific jobs. Induction heads (look back for the last time the current token appeared, copy what came after) are a clear example and underpin in-context pattern completion.
- Planning-ish behavior: there’s evidence that models sometimes pick a target (like a rhyme word) ahead of time and write toward it, rather than being purely myopic.
- Knowledge recall: factual lookups seem to involve attention heads moving subject info to a position, and MLP layers retrieving attributes associated with it.
- Unfaithful explanations: a model’s written explanation of how it got an answer doesn’t always match the computation it actually did. Don’t treat a chain of thought as a perfect window into the mechanism.
All of this is early-stage and incomplete. Anyone claiming “it’s just statistics, nothing happening inside” or “it definitely understands like a human” is overreaching in one direction or the other.
← 12. Inference · Contents · 14. Limitations and failure modes →