8. The transformer block

A modern LLM is a stack of identical transformer blocks (decoder-only variety), 30 to 120+ of them. Each block has the same structure:

flowchart TB
    IN["x  (input vector)"] --> N1["Norm"]
    N1 --> ATT["Attention<br/>tokens talk to each other"]
    ATT --> ADD1(("+"))
    IN ==>|"residual shortcut"| ADD1
    ADD1 --> N2["Norm"]
    N2 --> MLP["MLP<br/>each token thinks alone"]
    MLP --> ADD2(("+"))
    ADD1 ==>|"residual shortcut"| ADD2
    ADD2 --> OUT["x′  (output vector)"]

As equations (this is the “pre-norm” layout most modern models use):

$$\begin{array}{rl}{\mathbf{x}}^{\prime }& =\mathbf{x}+\text{Attention}(\text{Norm}(\mathbf{x}))\\ {\mathbf{x}}^{\prime \prime }& ={\mathbf{x}}^{\prime }+\text{MLP}(\text{Norm}({\mathbf{x}}^{\prime }))\end{array}$$

The vector for each token flows up through the blocks, getting refined at each step. Think of the residual stream as a shared bus that each block reads from and writes small updates to.

8a. Self-attention

The important bit

The problem it solves: to understand a token, you often need information from other tokens. In “The trophy didn’t fit in the suitcase because it was too big,” what “it” refers to depends on context. Attention is the mechanism that lets each token gather relevant info from other tokens.

How it works. For every token’s vector $\mathbf{x}$, we compute three different vectors using three learned matrices:

$\mathbf{q}=\mathbf{x}{W}_Q\mathbf{k}=\mathbf{x}{W}_K\mathbf{v}=\mathbf{x}{W}_V$

Vector	Name	Think of it as
$\mathbf{q}$	query	”What am I looking for?“
$\mathbf{k}$	key	”What do I contain / advertise?“
$\mathbf{v}$	value	”What info do I hand over if you attend to me?”

Then for a sequence, stacking all tokens into matrices $Q$, $K$, $V$:

$\boxed{\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}+M\right)V}$

Breaking down the attention formula, one symbol at a time

Piece	Name	What it does
$Q$, $K$, $V$	query, key, value matrices	One row per token. $n$ tokens → $n$ rows
$K^{\top }$	K transposed	Flip $K$ so rows become columns (so the multiply lines up)
$Q{K}^{\top }$	raw scores	Dot product of every query with every key → an $n\times n$ grid. Entry $(i,j)$ = “how relevant is token $j$ to token $i$?“
$\sqrt{d_k}$	scale factor	$d_k$ is the length of each key vector. Dividing by its square root keeps the scores from getting huge
$M$	causal mask	A grid of $0$s and $-\infty$s that blocks looking at the future
$\text{softmax}$	normalizer	Turns each row of scores into weights between 0 and 1 that sum to 1
$\cdots V$	weighted average	Each token’s output = weighted mix of all the value vectors

Step by step:

flowchart LR
    X["Token vectors X"] --> QKV["Make Q, K, V<br/>(3 matrix multiplies)"]
    QKV --> S["Scores<br/>Q·Kᵀ"]
    S --> SC["Scale<br/>÷ √d_k"]
    SC --> MK["Mask<br/>hide the future"]
    MK --> SM["Softmax<br/>rows sum to 1"]
    SM --> OUT["Weighted sum<br/>of V"]

1. $Q{K}^{\top }$: dot product of every token’s query with every token’s key. This gives an $n\times n$ matrix of raw relevance scores.

2. $÷\sqrt{d_k}$: scaling so the dot products don’t get huge and push softmax into saturated, near-zero-gradient territory.

3. $+M$ (the causal mask): when predicting the next token, a token may only look at itself and earlier tokens, never the future (otherwise it would just copy the answer during training). Future positions get $-\infty$ added so softmax makes them $0$:

   $M_{ij}=\{\begin{array}{ll}0& \text{if }j\le i\text{(past or present, allowed)}\\ -\infty & \text{if }j>i\text{(future, blocked)}\end{array}$

4. softmax: turns each row of scores into weights that sum to 1. (Full formula in Sampling.)

5. $\cdots V$: each token’s output is the weighted average of all the value vectors, using those weights.

What the causal mask looks like

For the 5-token sentence “The cat sat on mat”, a ✓ means “this row’s token is allowed to look at this column’s token”:

looks at →	The	cat	sat	on	mat
The	✓	·	·	·	·
cat	✓	✓	·	·	·
sat	✓	✓	✓	·	·
on	✓	✓	✓	✓	·
mat	✓	✓	✓	✓	✓

It’s a triangle. Every token sees itself and everything to its left, nothing to its right. The · cells are the $-\infty$ entries in $M$.

So attention is a learned, content-based, soft lookup. Each token asks a question (query), every earlier token offers a key, the best matches contribute their values.

Multi-head attention. Instead of one attention operation, run $h$ of them in parallel (e.g. 32 or 64 heads), each with its own smaller $W_Q,W_K,W_V$. Concatenate the results and mix with another matrix $W_O$:

$\text{MultiHead}(X)=\text{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W_O\text{where }{\text{head}}_i=\text{Attention}(X{W}_Q^{(i)},X{W}_K^{(i)},X{W}_V^{(i)})$

Different heads specialize in different things. Researchers have found heads that track syntax, heads that copy the previous occurrence of a token (induction heads, central to in-context learning), heads that attend to the previous token, and so on.

Cost note

The $n\times n$ score matrix means attention cost grows quadratically with sequence length ($n$ tokens → $n^2$ scores). That’s the main reason long context is expensive, and why a lot of engineering (FlashAttention, grouped-query attention, sliding windows) exists to cut the cost.

Try it

Watch attention run on real text, step by step, in 3D: Transformer Explainer and LLM Visualization.

8b. The MLP

After attention mixes information between tokens, the MLP (multi-layer perceptron, also called the feed-forward network) processes each token’s vector independently:

$\text{MLP}(\mathbf{x})=W_{\text{down}}\left(\text{act}\right(W_{\text{up}}\mathbf{x}\left)\right)$

Step	What happens
$W_{\text{up}}\mathbf{x}$	Expand the vector to a larger dimension (typically ~4× wider)
$\text{act}(\cdot )$	Apply the non-linearity (see Common activation functions)
$W_{\text{down}}(\cdot )$	Project back down to the original size

A gated variant like SwiGLU has a similar total size.

If attention is “communication between tokens,” the MLP is “computation within a token.” A big chunk of the model’s factual knowledge appears to be stored in these layers, acting like a fuzzy key-value memory. About two thirds of a transformer’s parameters live in the MLPs.

8c. Residual connections and normalization

Residual connection: each sublayer’s output is added to its input rather than replacing it:

$\mathbf{x}\leftarrow \mathbf{x}+\text{sublayer}(\mathbf{x})$

This lets gradients flow straight back through the network during backprop (fixing the vanishing gradient problem that made deep nets impractical), and it makes each layer an incremental refinement instead of a total rewrite.

Normalization (LayerNorm / RMSNorm): rescales the vector at various points to keep numbers in a stable range. Without it, activations blow up or shrink across dozens of layers and training falls apart. Modern models usually apply it before each sublayer (“pre-norm”).

RMSNorm formula, broken down

$\text{RMSNorm}(\mathbf{x}{)}_i=\frac{x_i}{\sqrt{\frac{1}{d}\sum _{j=1}^d{x}_j^2+\epsilon }}\cdot g_i$

Piece	Meaning
${\sum }_{j=1}^d{x}_j^2$	Square every entry, then add them all up
$\frac{1}{d}\sum \dots$	Divide by the count → the mean of the squares
$\sqrt{\dots }$	Square root → the “typical size” (root mean square, hence RMS)
$\epsilon$	Tiny number so we never divide by zero
$x_i/\sqrt{\dots }$	Divide each entry by the typical size → the vector now has a standard scale
$g_i$	A learned per-entry gain, so the model can re-scale if it wants

---

← 7. Embeddings · Contents · 9. Stacking it up and producing output →