The model works on tokens, not characters or words. A tokenizer splits text into pieces from a fixed vocabulary (typically 32k to 200k entries).
Most modern tokenizers use Byte Pair Encoding (BPE) or a close variant:
flowchart TB A["Step 1: Start with a vocabulary of<br/>individual bytes / characters"] --> B["Step 2: Count the most frequent<br/>adjacent pair in a big corpus"] B --> C["Step 3: Merge that pair into<br/>a new token"] C --> D{"Step 4: Repeated enough<br/>times?"} D -->|"no, tens of thousands of rounds"| B D -->|"yes"| E["Final vocabulary"]
Result: common words become single tokens (" the"), rarer words get split into pieces ("un" + "believ" + "able"), and anything weird falls back to smaller chunks or raw bytes, so nothing is ever “unknown.”
Consequences worth knowing
- Models don’t see letters. That’s why “how many r’s in strawberry” is historically a pain. The word is one or two opaque tokens.
- Arithmetic and spelling are harder than they look because digits and words get chopped inconsistently.
- Non-English text often costs more tokens per word, which means less effective context and higher cost.
- A rough rule for English: 1 token ≈ 0.75 words.
Each token maps to an integer ID: "cat" -> 9246, for example. That’s all the model gets.
Try it
See exactly how real tokenizers chop up your own text: Tiktokenizer.
← 5. From generic networks to language · Contents · 7. Embeddings →