How an LLM Actually Works (Full Version)

From a single neuron to a trained model

What this is

A bottom-up tour: neurons → networks → how networks learn → how that scales into something that talks. If you only read two sections, read 8. The transformer block and 10. Pretraining at scale. That’s where the real stuff is.

How to get the most out of this guide

• Click any link to jump to that section. Every formula links back to the Symbol cheat sheet.
• Hover a link to preview the section without leaving where you are.
• Collapsed boxes (marked with a small arrow) hold worked examples and extra detail. Click to expand.
• Diagrams are live Mermaid charts and equations are rendered LaTeX.
• Want to play with the ideas? Jump to Interactive visualizers.

---

Table of contents

#	Section	In one line
0	Symbol cheat sheet	What Σ, ∂, θ and friends actually mean
0	The big picture	One diagram of the whole pipeline
1	1. The artificial neuron	Multiply, add, squash
2	2. Layers and why non-linearity matters	Matrix multiplies and the “stacking” trick
3	3. Loss and how wrong we are	A single number that says “bad” or “good”
4	4. Gradient descent and backpropagation	How the weights actually improve
5	5. From generic networks to language	Two problems to solve for text
6	6. Tokenization	Text → integer IDs
7	7. Embeddings	IDs → meaningful vectors
8	8. The transformer block	Attention, MLP, residuals, norm
9	9. Stacking it up and producing output	Full forward pass
10	10. Pretraining at scale	Data, objective, scaling laws, hardware
11	11. Post-training	Turning a base model into an assistant
12	12. Inference	What happens when you hit enter
13	13. What’s actually going on inside	Interpretability
14	14. Limitations and failure modes	Where it breaks
15	15. Tiny code examples	The skeleton in ~50 lines
16	16. Where to learn more	Resources + Interactive visualizers

---

Symbol cheat sheet

"What's the sideways M?"

That’s Σ, the capital Greek letter sigma. It just means “add up a bunch of things.” It looks like an M lying on its side, but it’s a sum.

${\sum }_{i=1}^3{x}_i=x_1+x_2+x_3$

Read it as: “for i going from 1 to 3, take x sub i, and add them all up.” The number under the Σ is where you start, the number on top is where you stop. If $x=(2,5,7)$, the sum is $2+5+7=14$.

It’s the same idea as a for loop with a running total:

total = 0
for i in range(1, 4):
    total += x[i]

Every symbol used in this note, in one place:

Symbol	Say it as	What it means	Tiny example
$\sum$	”sum” (capital sigma)	Add up a list of terms	${\sum }_{i=1}^{3}i=1+2+3=6$
$\sigma$	”sigma” (lowercase)	The sigmoid function, squashes any number into 0 to 1. Different from Σ!	$\sigma (0)=0.5$
$\partial$	“partial"	"A tiny change in…”. Used in derivatives	$\frac{\partial L}{\partial w}$ = how much the loss moves when $w$ moves a hair
$\theta$	”theta”	Shorthand for all the model’s parameters (every weight and bias)	$p_{\theta }$ = probabilities from the model with parameters $\theta$
$\eta$	“eta”	The learning rate, i.e. the step size	$\eta =0.01$
$\epsilon$	”epsilon”	A tiny number added to avoid dividing by zero	$\epsilon =10^{-6}$
$\mathbf{x}$	”x vector”	A bold letter means a list of numbers (vector)	$\mathbf{x}=(2,-1,3)$
$W$	“W”	A capital letter usually means a grid of numbers (matrix)	A $3\times 4$ grid
$\mathbf{x}\cdot \mathbf{w}$	”dot product”	Multiply matching entries, then add them all up	$(1,2)\cdot (3,4)=3+8=11$
$\odot$	”elementwise times”	Multiply two lists entry-by-entry (no adding)	$(1,2)\odot (3,4)=(3,8)$
$K^{\top }$	“K transpose”	Flip a grid so rows become columns	A $5\times 8$ grid becomes $8\times 5$
$\in$	“is in”	Belongs to a set	$x\in \mathbb{R}$ = “x is a real number"
${\mathbb{R}}^d$	"R to the d”	A list of $d$ real numbers	${\mathbb{R}}^{4096}$ = a 4096-long vector
$x_i$	”x sub i”	The i-th entry of $x$ (the little number is just a label)	$x_2$ = second entry
$x_{<t}$	”x before t”	Everything that came before position $t$	For $t=4$: $x_1,x_2,x_3$
$\leftarrow$	“gets” / “becomes”	Overwrite the left side with the right side (assignment)	$w\leftarrow w-1$
$\approx$	“approximately”	Close to, not exactly equal	$\pi \approx 3.14$
$\sqrt{x}$	”square root”	The number that squared gives $x$	$\sqrt{16}=4$
$e^x$ or $\exp (x)$	“e to the x”	Exponential. $e\approx 2.718$. Always positive, grows fast	$\exp (0)=1$, $\exp (1)\approx 2.72$
$\log (x)$	“log”	Natural log, the inverse of $\exp$. Negative for $x<1$	$\log (1)=0$, $\log (0.5)\approx -0.69$
$\infty$	“infinity”	Bigger than any number	$-\infty$ is used to “turn off” things
$\Phi (x)$	”capital phi of x”	Probability that a standard bell-curve random number is below $x$ (used in GELU)	$\Phi (0)=0.5$

Two more reading tips

• $p(a\mid b)$ is read “probability of $a$ given $b$.” The tall bar means “given” or “knowing that.”
• A fraction like $\frac{\partial L}{\partial w}$ is just one symbol-group. Don’t try to cancel the $\partial$‘s like normal fractions.

↑ Back to contents

---

The big picture

Before the details, here is the whole machine on one page. Every box below gets its own section.

flowchart LR
    A["Raw text"] --> B["Tokenizer<br/>(section 6)"]
    B --> C["Embeddings<br/>(section 7)"]
    C --> D["Transformer blocks<br/>x L layers<br/>(section 8)"]
    D --> E["Unembedding<br/>(section 9)"]
    E --> F["Softmax<br/>probabilities"]
    F --> G["Sample next token<br/>(section 12)"]
    G -.->|"append and repeat"| A

And here is how it learns (sections 3 and 4):

flowchart LR
    A["Batch of text"] --> B["Forward pass<br/>make predictions"]
    B --> C["Loss<br/>how wrong?"]
    C --> D["Backward pass<br/>compute gradients"]
    D --> E["Update weights<br/>w ← w − η·gradient"]
    E --> A

↑ Back to contents

---

1. The artificial neuron

The “neuron” in a neural network is a loose, very lossy analogy to a biological one. Mathematically it’s almost embarrassingly simple: multiply each input by a weight, add them up, add a bias, then squash the result.

$y=f\left(w_1{x}_1+w_2{x}_2+\cdots +w_n{x}_n+b\right)$

Written compactly with the sum symbol (see Symbol cheat sheet):

$y=f\left({\sum }_{i=1}^n{w}_i{x}_i+b\right)$

And in vector notation, which is what you’ll see everywhere:

$y=f(\mathbf{w}\cdot \mathbf{x}+b)$

Breaking down the formula

Piece	Name	What it is
$x_1\dots x_n$	inputs	The numbers coming in
$w_1\dots w_n$	weights	Learned numbers saying how much each input matters
$b$	bias	A learned offset (lets the neuron fire even when inputs are zero)
${\sum }_{i=1}^n{w}_i{x}_i$	weighted sum	Multiply each input by its weight, then add all of them
$f$	activation function	A non-linear squashing or gating function
$y$	output	The single number the neuron produces

That dot product is the core operation of the whole field. Everything else is organization around it.

flowchart LR
    x1["x₁"] -->|"× w₁"| S(("Σ + b"))
    x2["x₂"] -->|"× w₂"| S
    x3["x₃"] -->|"× w₃"| S
    S --> F["f( ) activation"]
    F --> Y["output y"]

Worked example with real numbers

Inputs $\mathbf{x}=(2,-1,3)$, weights $\mathbf{w}=(0.5,1,0.2)$, bias $b=0.1$, activation = ReLU.

$$\begin{array}{rl}\text{weighted sum}& =(0.5)(2)+(1)(-1)+(0.2)(3)=1-1+0.6=0.6\\ \text{add bias}& =0.6+0.1=0.7\\ y& =\text{ReLU}(0.7)=\max (0,0.7)=0.7\end{array}$$

If the pre-activation had been negative (say $-0.5$), ReLU would output $0$. The neuron “stays off.”

Common activation functions

Name	Formula	Notes
ReLU	$\max (0,x)$	Dead simple, was the default for years. “If negative, output 0, else pass through.”
GELU	$x\cdot \Phi (x)$	A smooth ReLU-ish curve. Used in GPT-2/BERT era. $\Phi$ is defined in the Symbol cheat sheet.
SiLU / Swish	$x\cdot \sigma (x)$, where $\sigma (x)=\frac{1}{1+e^{-x}}$	Common in modern models. $\sigma$ (the sigmoid) squashes any number into the range 0 to 1.
SwiGLU	$(\text{SiLU}(x{W}_{\text{gate}})\odot x{W}_{\text{up}})W_{\text{down}}$	A gated variant: one path decides “how much to let through” for the other. What most modern LLMs use in their MLPs (see 8b. The MLP).

↑ Back to contents

---

2. Layers and why non-linearity matters

One neuron is useless alone. You stack many side by side to form a layer, and many layers to form a network.

A layer with 4 inputs and 3 neurons is just a matrix multiply:

$\mathbf{y}=f(W\mathbf{x}+\mathbf{b})$

Breaking down the formula

Piece	Shape	Meaning
$\mathbf{x}$	4 numbers	The inputs
$W$	$3\times 4$ grid	One row of weights per neuron
$W\mathbf{x}$	3 numbers	Each neuron’s weighted sum, all computed at once
$\mathbf{b}$	3 numbers	One bias per neuron
$f$		Applied to each of the 3 numbers separately
$\mathbf{y}$	3 numbers	The layer’s output

That’s why GPUs matter. A neural network is mostly giant matrix multiplications, and GPUs are built to do those in parallel.

Why the activation function is non-negotiable

If you remove $f$, then stacking layers is pointless:

$W_2(W_1\mathbf{x})=(W_2{W}_1)\mathbf{x}$

Two matrices multiplied together is just one matrix. Without non-linearity, a 100-layer network collapses into a single linear transformation and can only learn straight-line relationships. The non-linearity is what lets stacked layers represent complicated functions.

Universal approximation theorem: a network with enough neurons and a non-linearity can approximate basically any continuous function. This is a “this can work in principle” result. It says nothing about whether you can actually find the right weights, which is the training problem.

What "parameters" means

The weights $W$ and biases $\mathbf{b}$ across the whole network are the parameters. When someone says “a 70B model,” they mean 70 billion of these numbers. Those numbers are the model. Nothing else is stored: no rules, no database, no facts in a table. Just numbers.

Try it

Watch a tiny network carve up a 2D plane as you add neurons and layers: TensorFlow Playground.

↑ Back to contents

---

3. Loss and how wrong we are

To learn, the network needs a number telling it how badly it’s doing. That’s the loss function.

For language models the task is: given the text so far, predict the next token. The network outputs a probability for every possible token. If the true next token was “cat” and the model gave “cat” a probability of 0.02, that’s bad. If it gave 0.9, that’s good.

The loss used is cross-entropy (also called negative log-likelihood):

$L=-\log p(\text{correct token})$

Averaged over a whole text of $N$ tokens, this becomes:

$L=-\frac{1}{N}{\sum }_{t=1}^N\log p_{\theta }\left(x_t\mid x_{<t}\right)$

Breaking down the averaged formula, piece by piece

Piece	How to read it
${\sum }_{t=1}^N$	”For every position $t$ from 1 to $N$, add up…”
$x_t$	The actual token at position $t$
$x_{<t}$	All the tokens before position $t$ (the context the model gets to see)
$p_{\theta }(x_t\mid x_{<t})$	“The probability the model (with parameters $\theta$) assigned to the true token, given everything before it”
$\log (\cdot )$	Natural log. Turns a probability into a negative number (or 0 if perfect)
$-$ (leading minus)	Flips it so the loss is positive. Bigger = worse
$\frac{1}{N}$	Divide by the count, i.e. take the average

Plain English: “On average, how surprised was the model by the real next token?”

Model’s probability for the right token	Loss $-\log p$	Reaction
$p=1.0$	$0$	Perfect
$p=0.9$	$\approx 0.11$	Great
$p=0.5$	$\approx 0.69$	Meh
$p=0.02$	$\approx 3.9$	Bad
$p=0.01$	$\approx 4.6$	Very bad
$p\to 0$	$\to \infty$	Catastrophic

The log makes confident wrong answers get punished hard. Training means: adjust the parameters so this loss, averaged over a gigantic amount of text, goes down.

Perplexity

You’ll also see perplexity:

$\text{perplexity}=e^L$

Roughly “how many tokens is the model effectively choosing between at each step.” A perplexity of 10 means it’s about as unsure as picking uniformly from 10 options.

↑ Back to contents

---

4. Gradient descent and backpropagation

The goal

We have a loss $L$ that depends on billions of parameters. We want to find parameter values that make $L$ small. We can’t solve this analytically. So we do it iteratively.

Gradient descent

The gradient $\frac{\partial L}{\partial w}$ tells you, for each parameter, “if I increase this weight slightly, does the loss go up or down, and by how much?” The update rule:

$w\leftarrow w-\eta \frac{\partial L}{\partial w}$

Breaking down the formula

Piece	Meaning
$w$	One weight in the network
$\leftarrow$	”Replace the left side with the right side”
$\frac{\partial L}{\partial w}$	The slope: how much $L$ changes per tiny nudge of $w$. Positive means “increasing $w$ makes things worse”
$\eta$	Learning rate, the step size (e.g. $0.0003$)
$-$	Go the opposite way of the slope, i.e. downhill

Move each weight a small step in the direction that reduces the loss. Repeat. Imagine standing on a foggy hillside and always stepping downhill. You never see the whole landscape, you just feel the local slope.

Backpropagation

The question is how to compute $\partial L/\partial w$ for every one of billions of weights efficiently. Answer: the chain rule from calculus, applied cleverly.

A network is a chain of functions:

$L=\text{loss}\left({\text{layer}}_N\right(\dots {\text{layer}}_2({\text{layer}}_1(\mathbf{x}))\dots \left)\right)$

The chain rule says the derivative of a composition is the product of derivatives along the chain. For a weight $w_1$ that feeds through $h\to a\to y\to L$:

$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial y}\cdot \frac{\partial y}{\partial a}\cdot \frac{\partial a}{\partial h}\cdot \frac{\partial h}{\partial w_1}$

Think of it as a chain of “how much does this change affect the next thing,” multiplied together.

Tiny example. Say $y=w_2\cdot \text{ReLU}(w_{1}x)$ and loss $L=(y-\text{target}{)}^2$.

flowchart LR
    X["x"] --> H["h = w₁·x"]
    H --> A["a = ReLU(h)"]
    A --> Y["y = w₂·a"]
    Y --> L["L = (y − target)²"]

Forward pass (compute left to right, saving every value):

$$\begin{array}{rl}h& =w_{1}x\\ a& =\text{ReLU}(h)\\ y& =w_{2}a\\ L& =(y-\text{target}{)}^2\end{array}$$

Backward pass (walk right to left, reusing each piece):

$$\begin{array}{rlrl}\frac{\partial L}{\partial y}& =2(y-\text{target})& & \\ \frac{\partial L}{\partial w_2}& =\frac{\partial L}{\partial y}\cdot a& & \text{(because }y=w_{2}a\text{, so }\partial y/\partial w_2=a\text{)}\\ \frac{\partial L}{\partial a}& =\frac{\partial L}{\partial y}\cdot w_2& & \\ \frac{\partial L}{\partial h}& =\frac{\partial L}{\partial a}\cdot \{\begin{array}{ll}1& h>0\\ 0& h\le 0\end{array}& & \text{(ReLU’s derivative)}\\ \frac{\partial L}{\partial w_1}& =\frac{\partial L}{\partial h}\cdot x& & \end{array}$$

You run the forward pass (compute outputs, saving intermediate values), then the backward pass (walk the chain in reverse, reusing each computed piece). The reuse is the trick. The backward pass costs about 2× a forward pass regardless of how many parameters there are. That’s what makes training billion-parameter networks feasible at all.

Worked example with numbers (click to expand)

Let $x=2$, $w_1=0.5$, $w_2=3$, target $=4$.

Forward: $h=1$, $a=1$, $y=3$, $L=(3-4{)}^2=1$

Backward:

• $\partial L/\partial y=2(3-4)=-2$
• $\partial L/\partial w_2=-2\cdot 1=-2$
• $\partial L/\partial a=-2\cdot 3=-6$
• $\partial L/\partial h=-6\cdot 1=-6$ (since $h>0$)
• $\partial L/\partial w_1=-6\cdot 2=-12$

Update with $\eta =0.01$:

• $w_1\leftarrow 0.5-0.01(-12)=0.62$
• $w_2\leftarrow 3-0.01(-2)=3.02$

Check: new $h=1.24$, $a=1.24$, $y=3.74$, $L\approx 0.065$. The loss dropped from $1$ to about $0.065$ in a single step.

(Try $\eta =0.1$ and the weights overshoot past the target. That’s the “learning rate too high” problem in miniature.)

Modern frameworks (PyTorch, JAX) do this automatically (autograd). You write the forward pass, and loss.backward() hands you every gradient.

Practical optimizers

Plain gradient descent is rarely used. Real training uses:

• Mini-batches: compute the gradient on a batch of examples (millions of tokens), not the whole dataset. It’s a noisy estimate, but good enough and way cheaper. This is stochastic gradient descent.
• Adam / AdamW: keeps running averages of each parameter’s gradient (momentum) and squared gradient (scale), then adapts the step size per parameter. This is the standard for LLMs.
• Learning rate schedule: warm up from tiny, ramp to a peak, then decay (often cosine). Too high a rate and training diverges. Too low and it crawls.

Adam in formulas (simplified)

Let $g$ be the current gradient.

$$\begin{array}{rlrl}m& \leftarrow {\beta }_{1}m+(1-{\beta }_1)g& & \text{(running average of gradients = momentum)}\\ v& \leftarrow {\beta }_{2}v+(1-{\beta }_2)g^2& & \text{(running average of squared gradients = scale)}\\ w& \leftarrow w-\eta \frac{m}{\sqrt{v}+\epsilon }& & \text{(step: direction from }m\text{, size tamed by }\sqrt{v}\text{)}\end{array}$$

${\beta }_1\approx 0.9$ and ${\beta }_2\approx 0.95$ to $0.999$ control how long the “memory” lasts. Parameters with big, jumpy gradients get smaller steps; quiet ones get bigger steps. (The real algorithm also includes a small bias-correction term, omitted here.)

↑ Back to contents

---

5. From generic networks to language

Everything above works on any data. To make it work on text we need to solve two problems:

1. Text isn’t numbers. We need a way to feed it in. → 6. Tokenization + 7. Embeddings
2. Language depends on context across long distances. A plain stack of layers processes a fixed-size input and has no good mechanism for “word 5 depends on word 900.” → 8a. Self-attention

Older approaches (RNNs, LSTMs) processed text one token at a time with a hidden state carried forward. They struggled with long-range dependencies and couldn’t be parallelized across the sequence, so they didn’t scale. The 2017 paper “Attention Is All You Need” introduced the transformer, which handles the whole sequence in parallel and uses attention to move information between positions. That unlocked scaling, and scaling is what made LLMs happen.

↑ Back to contents

---

6. Tokenization

The model works on tokens, not characters or words. A tokenizer splits text into pieces from a fixed vocabulary (typically 32k to 200k entries).

Most modern tokenizers use Byte Pair Encoding (BPE) or a close variant:

flowchart TB
    A["Step 1: Start with a vocabulary of<br/>individual bytes / characters"] --> B["Step 2: Count the most frequent<br/>adjacent pair in a big corpus"]
    B --> C["Step 3: Merge that pair into<br/>a new token"]
    C --> D{"Step 4: Repeated enough<br/>times?"}
    D -->|"no, tens of thousands of rounds"| B
    D -->|"yes"| E["Final vocabulary"]

Result: common words become single tokens (" the"), rarer words get split into pieces ("un" + "believ" + "able"), and anything weird falls back to smaller chunks or raw bytes, so nothing is ever “unknown.”

Consequences worth knowing

• Models don’t see letters. That’s why “how many r’s in strawberry” is historically a pain. The word is one or two opaque tokens.
• Arithmetic and spelling are harder than they look because digits and words get chopped inconsistently.
• Non-English text often costs more tokens per word, which means less effective context and higher cost.
• A rough rule for English: 1 token ≈ 0.75 words.

Each token maps to an integer ID: "cat" -> 9246, for example. That’s all the model gets.

Try it

See exactly how real tokenizers chop up your own text: Tiktokenizer.

↑ Back to contents

---

7. Embeddings

An integer ID isn’t useful as a number (ID 9246 isn’t “bigger” than ID 12 in any meaningful way). So we use an embedding table: a big matrix with one row per token.

$E\in {\mathbb{R}}^{V\times d_{\text{model}}}$

Breaking down the notation

Piece	Meaning
$E$	The embedding table (a grid of learned numbers)
$\in$	“is a member of"
${\mathbb{R}}^{V\times d_{\text{model}}}$	"A grid of real numbers with $V$ rows and $d_{\text{model}}$ columns”
$V$	Vocabulary size (number of distinct tokens, e.g. 100,000)
$d_{\text{model}}$	Width of the model (e.g. 4096 or 8192). The length of each token’s vector

Token ID $i$ simply selects row $i$, giving a vector of $d_{\text{model}}$ floats. These vectors are learned parameters, initialized randomly and shaped by training.

What emerges: tokens used in similar contexts end up with similar vectors. Directions in the space come to encode concepts. The old famous example:

$\overrightarrow{\text{king}}-\overrightarrow{\text{man}}+\overrightarrow{\text{woman}}\approx \overrightarrow{\text{queen}}$

Nobody designed that. It’s what minimizes next-token loss.

Try it

Fly around a real embedding space in 3D: Embedding Projector.

Positional information

Attention by itself has no sense of order. It treats the input as a bag of vectors. So we inject position information. Original transformers added sinusoidal or learned position vectors. Most modern LLMs use RoPE (rotary position embeddings), which rotates the query and key vectors by an angle depending on position, so the attention score between two tokens naturally depends on how far apart they are.

↑ Back to contents

---

8. The transformer block

A modern LLM is a stack of identical transformer blocks (decoder-only variety), 30 to 120+ of them. Each block has the same structure:

flowchart TB
    IN["x  (input vector)"] --> N1["Norm"]
    N1 --> ATT["Attention<br/>tokens talk to each other"]
    ATT --> ADD1(("+"))
    IN ==>|"residual shortcut"| ADD1
    ADD1 --> N2["Norm"]
    N2 --> MLP["MLP<br/>each token thinks alone"]
    MLP --> ADD2(("+"))
    ADD1 ==>|"residual shortcut"| ADD2
    ADD2 --> OUT["x′  (output vector)"]

As equations (this is the “pre-norm” layout most modern models use):

$$\begin{array}{rl}{\mathbf{x}}^{\prime }& =\mathbf{x}+\text{Attention}(\text{Norm}(\mathbf{x}))\\ {\mathbf{x}}^{\prime \prime }& ={\mathbf{x}}^{\prime }+\text{MLP}(\text{Norm}({\mathbf{x}}^{\prime }))\end{array}$$

The vector for each token flows up through the blocks, getting refined at each step. Think of the residual stream as a shared bus that each block reads from and writes small updates to.

8a. Self-attention

The important bit

The problem it solves: to understand a token, you often need information from other tokens. In “The trophy didn’t fit in the suitcase because it was too big,” what “it” refers to depends on context. Attention is the mechanism that lets each token gather relevant info from other tokens.

How it works. For every token’s vector $\mathbf{x}$, we compute three different vectors using three learned matrices:

$\mathbf{q}=\mathbf{x}{W}_Q\mathbf{k}=\mathbf{x}{W}_K\mathbf{v}=\mathbf{x}{W}_V$

Vector	Name	Think of it as
$\mathbf{q}$	query	”What am I looking for?“
$\mathbf{k}$	key	”What do I contain / advertise?“
$\mathbf{v}$	value	”What info do I hand over if you attend to me?”

Then for a sequence, stacking all tokens into matrices $Q$, $K$, $V$:

$\boxed{\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}+M\right)V}$

Breaking down the attention formula, one symbol at a time

Piece	Name	What it does
$Q$, $K$, $V$	query, key, value matrices	One row per token. $n$ tokens → $n$ rows
$K^{\top }$	K transposed	Flip $K$ so rows become columns (so the multiply lines up)
$Q{K}^{\top }$	raw scores	Dot product of every query with every key → an $n\times n$ grid. Entry $(i,j)$ = “how relevant is token $j$ to token $i$?“
$\sqrt{d_k}$	scale factor	$d_k$ is the length of each key vector. Dividing by its square root keeps the scores from getting huge
$M$	causal mask	A grid of $0$s and $-\infty$s that blocks looking at the future
$\text{softmax}$	normalizer	Turns each row of scores into weights between 0 and 1 that sum to 1
$\cdots V$	weighted average	Each token’s output = weighted mix of all the value vectors

Step by step:

flowchart LR
    X["Token vectors X"] --> QKV["Make Q, K, V<br/>(3 matrix multiplies)"]
    QKV --> S["Scores<br/>Q·Kᵀ"]
    S --> SC["Scale<br/>÷ √d_k"]
    SC --> MK["Mask<br/>hide the future"]
    MK --> SM["Softmax<br/>rows sum to 1"]
    SM --> OUT["Weighted sum<br/>of V"]

1. $Q{K}^{\top }$: dot product of every token’s query with every token’s key. This gives an $n\times n$ matrix of raw relevance scores.

2. $÷\sqrt{d_k}$: scaling so the dot products don’t get huge and push softmax into saturated, near-zero-gradient territory.

3. $+M$ (the causal mask): when predicting the next token, a token may only look at itself and earlier tokens, never the future (otherwise it would just copy the answer during training). Future positions get $-\infty$ added so softmax makes them $0$:

   $M_{ij}=\{\begin{array}{ll}0& \text{if }j\le i\text{(past or present, allowed)}\\ -\infty & \text{if }j>i\text{(future, blocked)}\end{array}$

4. softmax: turns each row of scores into weights that sum to 1. (Full formula in Sampling.)

5. $\cdots V$: each token’s output is the weighted average of all the value vectors, using those weights.

What the causal mask looks like

For the 5-token sentence “The cat sat on mat”, a ✓ means “this row’s token is allowed to look at this column’s token”:

looks at →	The	cat	sat	on	mat
The	✓	·	·	·	·
cat	✓	✓	·	·	·
sat	✓	✓	✓	·	·
on	✓	✓	✓	✓	·
mat	✓	✓	✓	✓	✓

It’s a triangle. Every token sees itself and everything to its left, nothing to its right. The · cells are the $-\infty$ entries in $M$.

So attention is a learned, content-based, soft lookup. Each token asks a question (query), every earlier token offers a key, the best matches contribute their values.

Multi-head attention. Instead of one attention operation, run $h$ of them in parallel (e.g. 32 or 64 heads), each with its own smaller $W_Q,W_K,W_V$. Concatenate the results and mix with another matrix $W_O$:

$\text{MultiHead}(X)=\text{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W_O\text{where }{\text{head}}_i=\text{Attention}(X{W}_Q^{(i)},X{W}_K^{(i)},X{W}_V^{(i)})$

Different heads specialize in different things. Researchers have found heads that track syntax, heads that copy the previous occurrence of a token (induction heads, central to in-context learning), heads that attend to the previous token, and so on.

Cost note

The $n\times n$ score matrix means attention cost grows quadratically with sequence length ($n$ tokens → $n^2$ scores). That’s the main reason long context is expensive, and why a lot of engineering (FlashAttention, grouped-query attention, sliding windows) exists to cut the cost.

Try it

Watch attention run on real text, step by step, in 3D: Transformer Explainer and LLM Visualization.

8b. The MLP

After attention mixes information between tokens, the MLP (multi-layer perceptron, also called the feed-forward network) processes each token’s vector independently:

$\text{MLP}(\mathbf{x})=W_{\text{down}}\left(\text{act}\right(W_{\text{up}}\mathbf{x}\left)\right)$

Step	What happens
$W_{\text{up}}\mathbf{x}$	Expand the vector to a larger dimension (typically ~4× wider)
$\text{act}(\cdot )$	Apply the non-linearity (see Common activation functions)
$W_{\text{down}}(\cdot )$	Project back down to the original size

A gated variant like SwiGLU has a similar total size.

If attention is “communication between tokens,” the MLP is “computation within a token.” A big chunk of the model’s factual knowledge appears to be stored in these layers, acting like a fuzzy key-value memory. About two thirds of a transformer’s parameters live in the MLPs.

8c. Residual connections and normalization

Residual connection: each sublayer’s output is added to its input rather than replacing it:

$\mathbf{x}\leftarrow \mathbf{x}+\text{sublayer}(\mathbf{x})$

This lets gradients flow straight back through the network during backprop (fixing the vanishing gradient problem that made deep nets impractical), and it makes each layer an incremental refinement instead of a total rewrite.

Normalization (LayerNorm / RMSNorm): rescales the vector at various points to keep numbers in a stable range. Without it, activations blow up or shrink across dozens of layers and training falls apart. Modern models usually apply it before each sublayer (“pre-norm”).

RMSNorm formula, broken down

$\text{RMSNorm}(\mathbf{x}{)}_i=\frac{x_i}{\sqrt{\frac{1}{d}\sum _{j=1}^d{x}_j^2+\epsilon }}\cdot g_i$

Piece	Meaning
${\sum }_{j=1}^d{x}_j^2$	Square every entry, then add them all up
$\frac{1}{d}\sum \dots$	Divide by the count → the mean of the squares
$\sqrt{\dots }$	Square root → the “typical size” (root mean square, hence RMS)
$\epsilon$	Tiny number so we never divide by zero
$x_i/\sqrt{\dots }$	Divide each entry by the typical size → the vector now has a standard scale
$g_i$	A learned per-entry gain, so the model can re-scale if it wants

↑ Back to contents

---

9. Stacking it up and producing output

Full forward pass for a sequence of tokens:

flowchart TB
    A["Step 1: Tokenize text<br/>→ t₁, t₂, …, tₙ"] --> B["Step 2: Embedding lookup (+ positions)<br/>→ n vectors of size d_model"]
    B --> C["Step 3: Repeat L times:<br/>attention block (+ residual)<br/>MLP block (+ residual)"]
    C --> D["Step 4: Final norm"]
    D --> E["Step 5: Unembedding<br/>multiply by a d_model × vocab matrix<br/>→ a logit for every token, per position"]
    E --> F["Step 6: Softmax over logits<br/>→ probability distribution over next token"]

The unembedding and softmax steps in math:

$\text{logits}=\mathbf{h}{W}_U{p}_i=\frac{e^{z_i}}{\sum _{j=1}^V{e}^{z_j}}$

Here $\mathbf{h}$ is the final vector for a position, $W_U$ is the $d_{\text{model}}\times V$ unembedding matrix, $z_i$ is the logit (raw score) for token $i$, and $p_i$ is the resulting probability. See Sampling for what that softmax is doing.

Key detail

During training, the model outputs a prediction at every position simultaneously. Position 1 predicts token 2, position 2 predicts token 3, and so on. A single sequence of 4,000 tokens gives 4,000 training signals in one forward pass. That’s a huge part of why the training is so efficient. The causal mask guarantees no position can cheat by peeking ahead.

Many models tie the embedding and unembedding matrices (same weights used for both), which saves parameters.

Counting parameters

Rough rule: each block has about $12d_{\text{model}}^2$ parameters (4 $d^2$ in attention, 8 $d^2$ in the MLP). With $L$ layers:

$\text{params}\approx 12\cdot L\cdot d_{\text{model}}^2$

For GPT-3: $d_{\text{model}}=12,288$ and $L=96$ layers.

$12\times 96\times (12,288{)}^2\approx 174\text{ billion}(\text{reported: }175\text{B})$

The math checks out, which is a nice sanity test that you understand the architecture.

↑ Back to contents

---

10. Pretraining at scale

The data

Trillions of tokens: filtered web crawl, books, academic papers, code, forums, Wikipedia, and increasingly synthetic data. A huge amount of effort goes into data quality: deduplication, filtering junk, balancing domains. Data curation is arguably as important as architecture.

The objective

Just next-token prediction with cross-entropy (the formula from 3. Loss and how wrong we are). That’s it. Take a chunk of text (say 8k tokens), run it through, compute the average loss over all positions, backprop, update.

The training loop

for step in range(hundreds_of_thousands):
    batch = sample many sequences from dataset
    logits = model(batch.inputs)
    loss   = cross_entropy(logits, batch.targets)   # targets = inputs shifted by 1
    loss.backward()                                  # backprop
    clip gradients                                   # stability
    optimizer.step()                                 # AdamW update
    scheduler.step()                                 # adjust learning rate

Why this teaches anything beyond autocomplete

To predict the next token well across everything humans write, the model gets forced to learn whatever internal structure reduces loss:

• Grammar and syntax (obviously helps)
• Facts about the world (“The capital of France is ___”)
• Code semantics (to predict the next line of code, track variables and types)
• Arithmetic patterns, logic, causal structure of stories
• Models of who is speaking and what they likely know or want

Nobody hard-codes any of it. It’s all pressure from the loss function.

The single most important idea

A simple objective, at scale, forces rich internal representations.

Scaling laws

Empirically, loss falls as a smooth power law in three things: parameters ($N$), training tokens ($D$), and compute ($C$). This predictability is why labs can plan billion-dollar training runs.

The Chinchilla paper (2022) found that for a fixed compute budget, models were being trained on too little data, with roughly 20 tokens per parameter being compute-optimal:

$D\approx 20N$

In practice, modern models get trained on far more than that (an overtrained smaller model is cheaper to run, and inference cost dominates over a model’s lifetime).

Training compute is roughly:

$C\approx 6ND\text{FLOPs}$

(FLOPs = floating-point operations, i.e. individual multiplies and adds.) For a 70B model on 15T tokens:

$6\times (70\times 10^9)\times (15\times 10^{12})\approx 6.3\times 10^{24}\text{ FLOPs}$

That’s thousands of GPUs running for weeks to months.

How it physically runs

No single GPU holds the model plus its optimizer state. So training is split across thousands of chips:

Technique	What’s split up
Data parallelism	Each GPU group gets different batches; gradients get averaged
Tensor parallelism	Individual matrix multiplies are split across GPUs
Pipeline parallelism	Different layers live on different GPUs
Sharding (ZeRO / FSDP)	Optimizer state and weights are partitioned across devices
Mixed precision	Compute in bf16/fp8 for speed, keep master weights in fp32 for accuracy

A lot of the actual difficulty in frontier training is plain infrastructure: hardware fails constantly at this scale, loss spikes happen, and runs need checkpointing and recovery.

Emergence

Some abilities (multi-step arithmetic, in-context learning, instruction following to a degree) seem to appear fairly abruptly as models scale. There’s real debate over how much of this is true phase-change behavior versus an artifact of how we measure it. Be skeptical of anyone who claims to fully understand it.

↑ Back to contents

---

11. Post-training

Making it an assistant. A pretrained “base model” is an internet simulator. Give it a question and it might continue with more questions, a forum reply, or a quiz answer key. To get an assistant, you do more training.

flowchart LR
    A["Base model<br/>(internet simulator)"] --> B["Supervised<br/>fine-tuning (SFT)"]
    B --> C["Preference optimization<br/>(RLHF and relatives)"]
    C --> D["RL on verifiable tasks<br/>(reasoning)"]
    D --> E["Assistant"]
    E -.-> F["Tool use"]

Supervised fine-tuning (SFT)

Train on a (relatively) small set of high-quality prompt/response pairs written by humans or generated and filtered. Same next-token loss, but now on assistant-style conversations, usually with loss computed only on the response. This teaches the format and the basic behavior of “answer the person.”

Preference optimization (RLHF and relatives)

1. Sample multiple responses to a prompt.
2. Humans (or an AI model following written principles, in RLAIF / Constitutional-style setups) rank them.
3. Train a reward model to predict those preferences, or skip the explicit reward model and optimize directly (DPO and variants).
4. Use reinforcement learning (PPO, GRPO, etc.) to shift the model toward responses that score higher, usually with a penalty for drifting too far from the original model.

This is where much of the tone, helpfulness, refusal behavior, and “personality” comes from. It’s also where problems like sycophancy come from: if raters prefer answers that flatter them, the model learns to flatter.

Reinforcement learning on verifiable tasks

Newer “reasoning” models are trained with RL on problems where the answer can be automatically checked: math, code with unit tests, puzzles. The model generates a long chain of thought, gets rewarded if the final answer is right, and learns strategies like breaking problems down, checking work, and backtracking. The reasoning text it produces is a real computation aid: more tokens means more serial computation the model can use, since each token generated gets processed by every layer again.

Tool use

Models get trained to emit structured calls (search, run code, call an API). Outside software executes the call and pastes the result back into the context. The model itself never “does” anything. It outputs text, and a wrapper loop acts on it.

sequenceDiagram
    participant U as You
    participant M as Model
    participant W as Wrapper software
    participant T as Tool (search, code, API)
    U->>M: Question
    M->>W: Text that is a tool call
    W->>T: Runs the call
    T-->>W: Result
    W->>M: Pastes result into context
    M->>U: Final answer

↑ Back to contents

---

12. Inference

What happens when you hit enter.

1. Your prompt (plus the system prompt, conversation history, any retrieved docs or memory notes) is concatenated into one long token sequence.
2. Prefill: the whole prompt runs through the network in parallel. For every layer, the key and value vectors for every prompt token are computed and cached (the KV cache).
3. The final position’s output gives a probability distribution over the next token.
4. Sampling picks a token (see Sampling).
5. Decode loop: append that token, run only the new token through the network (it attends to the cached keys/values of everything before it), get the next distribution, sample, repeat. Stop on an end-of-sequence token or a length limit.

flowchart TB
    P["Prompt tokens"] --> PF["Prefill<br/>process all at once, fill KV cache"]
    PF --> D["Distribution over next token"]
    D --> S["Sample a token"]
    S --> Q{"End token or<br/>length limit?"}
    Q -->|"no"| N["Run only the new token<br/>(reuse KV cache)"]
    N --> D
    Q -->|"yes"| DONE["Done"]

Why the KV cache matters

Without it you’d recompute attention over the entire history for every new token. With it, each new token costs one pass of “new stuff only.” The tradeoff is memory: the cache grows with context length, and for long contexts it can be larger than the model weights. Decoding is typically memory-bandwidth bound (loading weights and cache from memory dominates), not compute bound.

Sampling

The model outputs a raw score called a logit, $z_i$, for each token. Turn them into probabilities with softmax, adjusted by a temperature $T$:

$\boxed{p_i=\frac{\exp (z_i/T)}{\sum _j\exp (z_j/T)}}$

Breaking down the formula (this is the one with the Σ)

Piece	What it does
$z_i$	The raw score (“logit”) the model gave to token $i$. Can be any number, positive or negative
$÷T$	Divide by temperature. Small $T$ exaggerates differences, large $T$ shrinks them
$\exp (\cdot )$	Raise $e$ to that power. This makes every score positive and magnifies gaps between scores
${\sum }_j\exp (z_j/T)$	The sum symbol: do the previous step for every token $j$ in the vocabulary, then add all those results together. This is the “total”
$\frac{\exp (z_i/T)}{{\sum }_j\cdots }$	Token $i$‘s share of the total, i.e. a probability between 0 and 1. All shares add up to 1

Plain English: exponentiate every score, then divide each by the grand total so they become percentages.

Worked example: three tokens, three temperatures

Suppose the model’s logits are $z=(2,1,0)$ for tokens A, B, C.

Temperature	A	B	C	Behavior
$T=0.5$	86.7%	11.7%	1.6%	Sharp, A dominates
$T=1$	66.5%	24.5%	9.0%	The model’s “true” distribution
$T=2$	50.6%	30.7%	18.6%	Flat, more random

Step-by-step for $T=1$:

1. $\exp (2)=7.389$, $\exp (1)=2.718$, $\exp (0)=1$
2. Total: $7.389+2.718+1=11.107$
3. A: $7.389/11.107=0.665$ ✓

Temperature	Effect
$T\to 0$	Always pick the top token (greedy, deterministic, can get repetitive)
$T=1$	Sample from the model’s actual distribution
$T>1$	Flatter distribution, more random and weirder

Common truncation tricks: top-k (only consider the $k$ most likely tokens), top-p / nucleus (smallest set of tokens whose probabilities sum to $p$). This sampling randomness is why the same prompt gives different outputs.

The context window

The model’s only “memory” within a conversation is the token sequence it’s fed. The context window is the maximum length of that sequence. When the conversation exceeds it, something has to get dropped or summarized. “Memory” features in chat products are just saved text that gets inserted into the prompt. The weights don’t change when you talk to the model.

↑ Back to contents

---

13. What’s actually going on inside

Honest state of knowledge

We fully understand the architecture and the training procedure. We do not fully understand the algorithms the trained network has learned. Training produces billions of numbers, and reverse-engineering them is its own research field (mechanistic interpretability).

Things researchers have actually found:

• Features: the residual stream encodes many human-interpretable concepts as directions in activation space. Because there are more concepts than dimensions, the model packs them in overlapping (superposition). Sparse autoencoders are a tool used to pull them apart into individual features.
• Circuits: identifiable sub-networks that do specific jobs. Induction heads (look back for the last time the current token appeared, copy what came after) are a clear example and underpin in-context pattern completion.
• Planning-ish behavior: there’s evidence that models sometimes pick a target (like a rhyme word) ahead of time and write toward it, rather than being purely myopic.
• Knowledge recall: factual lookups seem to involve attention heads moving subject info to a position, and MLP layers retrieving attributes associated with it.
• Unfaithful explanations: a model’s written explanation of how it got an answer doesn’t always match the computation it actually did. Don’t treat a chain of thought as a perfect window into the mechanism.

All of this is early-stage and incomplete. Anyone claiming “it’s just statistics, nothing happening inside” or “it definitely understands like a human” is overreaching in one direction or the other.

↑ Back to contents

---

14. Limitations and failure modes

Failure mode	What’s going on
Hallucination	The objective rewards plausible continuations, not true ones. When the model lacks the knowledge, it can still produce fluent, confident, wrong text. Post-training reduces this but doesn’t eliminate it.
Knowledge cutoff	Weights are frozen after training. Anything after the data cutoff is unknown unless provided in context or via tools.
Tokenization artifacts	Letter counting, exact string manipulation, some arithmetic (see 6. Tokenization).
Context limits and degradation	Performance can drop on very long inputs, especially for info buried in the middle.
Sycophancy	Tendency to agree with the user because that’s what preference training rewarded (see 11. Post-training).
Prompt injection	The model can’t fundamentally separate “instructions” from “data” in its context. Text in a webpage or file can try to hijack it.
Brittleness and distribution shift	Weird inputs far from the training distribution can produce weird behavior.
Non-determinism	Sampling plus floating-point and batching effects mean you won’t get identical outputs every time.
Inherited bias	The data reflects the internet, with all that implies.

↑ Back to contents

---

15. Tiny code examples

One attention head in plain NumPy

This is the attention formula from 8a. Self-attention, line for line:

import numpy as np
 
def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)   # numerical stability
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)
 
def causal_self_attention(X, W_Q, W_K, W_V):
    # X: [n, d_model]   W_*: [d_model, d_head]
    Q, K, V = X @ W_Q, X @ W_K, X @ W_V
    d_k = Q.shape[-1]
    scores = (Q @ K.T) / np.sqrt(d_k)              # [n, n]
    mask = np.triu(np.ones_like(scores), k=1)      # 1s above the diagonal = future
    scores = np.where(mask == 1, -1e9, scores)     # block looking ahead
    weights = softmax(scores, axis=-1)             # each row sums to 1
    return weights @ V                             # [n, d_head]
 
# toy run
n, d_model, d_head = 5, 16, 8
rng = np.random.default_rng(0)
X = rng.normal(size=(n, d_model))
out = causal_self_attention(
    X,
    rng.normal(size=(d_model, d_head)),
    rng.normal(size=(d_model, d_head)),
    rng.normal(size=(d_model, d_head)),
)
print(out.shape)  # (5, 8)

The training loop shape in PyTorch

import torch
import torch.nn.functional as F
 
model = MyTinyGPT(...)                       # embeddings -> blocks -> unembedding
opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
 
for step, (x, y) in enumerate(loader):       # y is x shifted left by one token
    logits = model(x)                        # [batch, seq, vocab]
    loss = F.cross_entropy(
        logits.view(-1, logits.size(-1)),
        y.view(-1),
    )
    opt.zero_grad()
    loss.backward()                          # backprop through everything
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step()

Generation loop

@torch.no_grad()
def generate(model, tokens, max_new=100, temperature=0.8):
    for _ in range(max_new):
        logits = model(tokens)[:, -1, :]                 # last position only
        probs = F.softmax(logits / temperature, dim=-1)
        next_tok = torch.multinomial(probs, num_samples=1)
        tokens = torch.cat([tokens, next_tok], dim=1)
    return tokens

That’s genuinely the skeleton. Production systems add KV caching, batching, better sampling, and a mountain of engineering, but the core idea fits on a page.

↑ Back to contents

---

16. Where to learn more

• Andrej Karpathy, “Neural Networks: Zero to Hero” (YouTube). Builds backprop, then a GPT, from scratch. Best single resource. makemore, nanoGPT, and “Let’s build GPT” are the key ones.
• 3Blue1Brown, deep learning series. Best visual intuition for neurons, backprop, and attention.
• “Attention Is All You Need” (Vaswani et al., 2017). The original transformer paper. Readable.
• Anthropic’s interpretability work (“A Mathematical Framework for Transformer Circuits,” “Toy Models of Superposition,” the sparse autoencoder / feature papers). For the “what’s inside” story.
• Chinchilla paper (Hoffmann et al., 2022) and the Kaplan scaling laws paper (2020). For scaling.
• Karpathy’s “Deep Dive into LLMs like ChatGPT” talk. Good end-to-end overview of pretraining through RL.

Interactive visualizers

Click to open in your browser and play with the ideas from this note:

Topic	Tool	What you can do
Neurons, layers, training (1. The artificial neuron, 2. Layers and why non-linearity matters)	TensorFlow Playground	Add neurons and layers, watch a network learn to separate points in real time
Tokenization (6. Tokenization)	Tiktokenizer	Paste any text and see exactly how it’s split into tokens and IDs
Embeddings (7. Embeddings)	Embedding Projector	Explore word vectors in 3D, find nearest neighbors
Attention and the full transformer (8. The transformer block)	Transformer Explainer	Run a small GPT-2 in your browser and watch attention and sampling temperature live
The whole model, layer by layer (9. Stacking it up and producing output)	LLM Visualization	A 3D walkthrough of every operation in a tiny GPT

If you want to actually internalize this

Write a tiny GPT yourself, train it on a text file, and watch the loss go down and the output go from garbage to vaguely coherent. That experience makes everything above click.

↑ Back to contents