12. Inference

What happens when you hit enter.

1. Your prompt (plus the system prompt, conversation history, any retrieved docs or memory notes) is concatenated into one long token sequence.
2. Prefill: the whole prompt runs through the network in parallel. For every layer, the key and value vectors for every prompt token are computed and cached (the KV cache).
3. The final position’s output gives a probability distribution over the next token.
4. Sampling picks a token (see Sampling).
5. Decode loop: append that token, run only the new token through the network (it attends to the cached keys/values of everything before it), get the next distribution, sample, repeat. Stop on an end-of-sequence token or a length limit.

flowchart TB
    P["Prompt tokens"] --> PF["Prefill<br/>process all at once, fill KV cache"]
    PF --> D["Distribution over next token"]
    D --> S["Sample a token"]
    S --> Q{"End token or<br/>length limit?"}
    Q -->|"no"| N["Run only the new token<br/>(reuse KV cache)"]
    N --> D
    Q -->|"yes"| DONE["Done"]

Why the KV cache matters

Without it you’d recompute attention over the entire history for every new token. With it, each new token costs one pass of “new stuff only.” The tradeoff is memory: the cache grows with context length, and for long contexts it can be larger than the model weights. Decoding is typically memory-bandwidth bound (loading weights and cache from memory dominates), not compute bound.

Sampling

The model outputs a raw score called a logit, $z_i$, for each token. Turn them into probabilities with softmax, adjusted by a temperature $T$:

$\boxed{p_i=\frac{\exp (z_i/T)}{\sum _j\exp (z_j/T)}}$

Breaking down the formula (this is the one with the Σ)

Piece	What it does
$z_i$	The raw score (“logit”) the model gave to token $i$. Can be any number, positive or negative
$÷T$	Divide by temperature. Small $T$ exaggerates differences, large $T$ shrinks them
$\exp (\cdot )$	Raise $e$ to that power. This makes every score positive and magnifies gaps between scores
${\sum }_j\exp (z_j/T)$	The sum symbol: do the previous step for every token $j$ in the vocabulary, then add all those results together. This is the “total”
$\frac{\exp (z_i/T)}{{\sum }_j\cdots }$	Token $i$‘s share of the total, i.e. a probability between 0 and 1. All shares add up to 1

Plain English: exponentiate every score, then divide each by the grand total so they become percentages.

Worked example: three tokens, three temperatures

Suppose the model’s logits are $z=(2,1,0)$ for tokens A, B, C.

Temperature	A	B	C	Behavior
$T=0.5$	86.7%	11.7%	1.6%	Sharp, A dominates
$T=1$	66.5%	24.5%	9.0%	The model’s “true” distribution
$T=2$	50.6%	30.7%	18.6%	Flat, more random

Step-by-step for $T=1$:

1. $\exp (2)=7.389$, $\exp (1)=2.718$, $\exp (0)=1$
2. Total: $7.389+2.718+1=11.107$
3. A: $7.389/11.107=0.665$ ✓

Temperature	Effect
$T\to 0$	Always pick the top token (greedy, deterministic, can get repetitive)
$T=1$	Sample from the model’s actual distribution
$T>1$	Flatter distribution, more random and weirder

Common truncation tricks: top-k (only consider the $k$ most likely tokens), top-p / nucleus (smallest set of tokens whose probabilities sum to $p$). This sampling randomness is why the same prompt gives different outputs.

The context window

The model’s only “memory” within a conversation is the token sequence it’s fed. The context window is the maximum length of that sequence. When the conversation exceeds it, something has to get dropped or summarized. “Memory” features in chat products are just saved text that gets inserted into the prompt. The weights don’t change when you talk to the model.

---

← 11. Post-training · Contents · 13. What’s actually going on inside →