Making it an assistant. A pretrained “base model” is an internet simulator. Give it a question and it might continue with more questions, a forum reply, or a quiz answer key. To get an assistant, you do more training.
flowchart LR A["Base model<br/>(internet simulator)"] --> B["Supervised<br/>fine-tuning (SFT)"] B --> C["Preference optimization<br/>(RLHF and relatives)"] C --> D["RL on verifiable tasks<br/>(reasoning)"] D --> E["Assistant"] E -.-> F["Tool use"]
Supervised fine-tuning (SFT)
Train on a (relatively) small set of high-quality prompt/response pairs written by humans or generated and filtered. Same next-token loss, but now on assistant-style conversations, usually with loss computed only on the response. This teaches the format and the basic behavior of “answer the person.”
Preference optimization (RLHF and relatives)
- Sample multiple responses to a prompt.
- Humans (or an AI model following written principles, in RLAIF / Constitutional-style setups) rank them.
- Train a reward model to predict those preferences, or skip the explicit reward model and optimize directly (DPO and variants).
- Use reinforcement learning (PPO, GRPO, etc.) to shift the model toward responses that score higher, usually with a penalty for drifting too far from the original model.
This is where much of the tone, helpfulness, refusal behavior, and “personality” comes from. It’s also where problems like sycophancy come from: if raters prefer answers that flatter them, the model learns to flatter.
Reinforcement learning on verifiable tasks
Newer “reasoning” models are trained with RL on problems where the answer can be automatically checked: math, code with unit tests, puzzles. The model generates a long chain of thought, gets rewarded if the final answer is right, and learns strategies like breaking problems down, checking work, and backtracking. The reasoning text it produces is a real computation aid: more tokens means more serial computation the model can use, since each token generated gets processed by every layer again.
Tool use
Models get trained to emit structured calls (search, run code, call an API). Outside software executes the call and pastes the result back into the context. The model itself never “does” anything. It outputs text, and a wrapper loop acts on it.
sequenceDiagram participant U as You participant M as Model participant W as Wrapper software participant T as Tool (search, code, API) U->>M: Question M->>W: Text that is a tool call W->>T: Runs the call T-->>W: Result W->>M: Pastes result into context M->>U: Final answer