7. Embeddings

An integer ID isn’t useful as a number (ID 9246 isn’t “bigger” than ID 12 in any meaningful way). So we use an embedding table: a big matrix with one row per token.

$E\in {\mathbb{R}}^{V\times d_{\text{model}}}$

Breaking down the notation

Piece	Meaning
$E$	The embedding table (a grid of learned numbers)
$\in$	“is a member of"
${\mathbb{R}}^{V\times d_{\text{model}}}$	"A grid of real numbers with $V$ rows and $d_{\text{model}}$ columns”
$V$	Vocabulary size (number of distinct tokens, e.g. 100,000)
$d_{\text{model}}$	Width of the model (e.g. 4096 or 8192). The length of each token’s vector

Token ID $i$ simply selects row $i$, giving a vector of $d_{\text{model}}$ floats. These vectors are learned parameters, initialized randomly and shaped by training.

What emerges: tokens used in similar contexts end up with similar vectors. Directions in the space come to encode concepts. The old famous example:

$\overrightarrow{\text{king}}-\overrightarrow{\text{man}}+\overrightarrow{\text{woman}}\approx \overrightarrow{\text{queen}}$

Nobody designed that. It’s what minimizes next-token loss.

Try it

Fly around a real embedding space in 3D: Embedding Projector.

Positional information

Attention by itself has no sense of order. It treats the input as a bag of vectors. So we inject position information. Original transformers added sinusoidal or learned position vectors. Most modern LLMs use RoPE (rotary position embeddings), which rotates the query and key vectors by an angle depending on position, so the attention score between two tokens naturally depends on how far apart they are.

---

← 6. Tokenization · Contents · 8. The transformer block →