4. Gradient descent and backpropagation

The goal

We have a loss $L$ that depends on billions of parameters. We want to find parameter values that make $L$ small. We can’t solve this analytically. So we do it iteratively.

Gradient descent

The gradient $\frac{\partial L}{\partial w}$ tells you, for each parameter, “if I increase this weight slightly, does the loss go up or down, and by how much?” The update rule:

$w\leftarrow w-\eta \frac{\partial L}{\partial w}$

Breaking down the formula

Piece	Meaning
$w$	One weight in the network
$\leftarrow$	”Replace the left side with the right side”
$\frac{\partial L}{\partial w}$	The slope: how much $L$ changes per tiny nudge of $w$. Positive means “increasing $w$ makes things worse”
$\eta$	Learning rate, the step size (e.g. $0.0003$)
$-$	Go the opposite way of the slope, i.e. downhill

Move each weight a small step in the direction that reduces the loss. Repeat. Imagine standing on a foggy hillside and always stepping downhill. You never see the whole landscape, you just feel the local slope.

Backpropagation

The question is how to compute $\partial L/\partial w$ for every one of billions of weights efficiently. Answer: the chain rule from calculus, applied cleverly.

A network is a chain of functions:

$L=\text{loss}\left({\text{layer}}_N\right(\dots {\text{layer}}_2({\text{layer}}_1(\mathbf{x}))\dots \left)\right)$

The chain rule says the derivative of a composition is the product of derivatives along the chain. For a weight $w_1$ that feeds through $h\to a\to y\to L$:

$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial y}\cdot \frac{\partial y}{\partial a}\cdot \frac{\partial a}{\partial h}\cdot \frac{\partial h}{\partial w_1}$

Think of it as a chain of “how much does this change affect the next thing,” multiplied together.

Tiny example. Say $y=w_2\cdot \text{ReLU}(w_{1}x)$ and loss $L=(y-\text{target}{)}^2$.

flowchart LR
    X["x"] --> H["h = w₁·x"]
    H --> A["a = ReLU(h)"]
    A --> Y["y = w₂·a"]
    Y --> L["L = (y − target)²"]

Forward pass (compute left to right, saving every value):

$$\begin{array}{rl}h& =w_{1}x\\ a& =\text{ReLU}(h)\\ y& =w_{2}a\\ L& =(y-\text{target}{)}^2\end{array}$$

Backward pass (walk right to left, reusing each piece):

$$\begin{array}{rlrl}\frac{\partial L}{\partial y}& =2(y-\text{target})& & \\ \frac{\partial L}{\partial w_2}& =\frac{\partial L}{\partial y}\cdot a& & \text{(because }y=w_{2}a\text{, so }\partial y/\partial w_2=a\text{)}\\ \frac{\partial L}{\partial a}& =\frac{\partial L}{\partial y}\cdot w_2& & \\ \frac{\partial L}{\partial h}& =\frac{\partial L}{\partial a}\cdot \{\begin{array}{ll}1& h>0\\ 0& h\le 0\end{array}& & \text{(ReLU’s derivative)}\\ \frac{\partial L}{\partial w_1}& =\frac{\partial L}{\partial h}\cdot x& & \end{array}$$

You run the forward pass (compute outputs, saving intermediate values), then the backward pass (walk the chain in reverse, reusing each computed piece). The reuse is the trick. The backward pass costs about 2× a forward pass regardless of how many parameters there are. That’s what makes training billion-parameter networks feasible at all.

Worked example with numbers (click to expand)

Let $x=2$, $w_1=0.5$, $w_2=3$, target $=4$.

Forward: $h=1$, $a=1$, $y=3$, $L=(3-4{)}^2=1$

Backward:

• $\partial L/\partial y=2(3-4)=-2$
• $\partial L/\partial w_2=-2\cdot 1=-2$
• $\partial L/\partial a=-2\cdot 3=-6$
• $\partial L/\partial h=-6\cdot 1=-6$ (since $h>0$)
• $\partial L/\partial w_1=-6\cdot 2=-12$

Update with $\eta =0.01$:

• $w_1\leftarrow 0.5-0.01(-12)=0.62$
• $w_2\leftarrow 3-0.01(-2)=3.02$

Check: new $h=1.24$, $a=1.24$, $y=3.74$, $L\approx 0.065$. The loss dropped from $1$ to about $0.065$ in a single step.

(Try $\eta =0.1$ and the weights overshoot past the target. That’s the “learning rate too high” problem in miniature.)

Modern frameworks (PyTorch, JAX) do this automatically (autograd). You write the forward pass, and loss.backward() hands you every gradient.

Practical optimizers

Plain gradient descent is rarely used. Real training uses:

• Mini-batches: compute the gradient on a batch of examples (millions of tokens), not the whole dataset. It’s a noisy estimate, but good enough and way cheaper. This is stochastic gradient descent.
• Adam / AdamW: keeps running averages of each parameter’s gradient (momentum) and squared gradient (scale), then adapts the step size per parameter. This is the standard for LLMs.
• Learning rate schedule: warm up from tiny, ramp to a peak, then decay (often cosine). Too high a rate and training diverges. Too low and it crawls.

Adam in formulas (simplified)

Let $g$ be the current gradient.

$$\begin{array}{rlrl}m& \leftarrow {\beta }_{1}m+(1-{\beta }_1)g& & \text{(running average of gradients = momentum)}\\ v& \leftarrow {\beta }_{2}v+(1-{\beta }_2)g^2& & \text{(running average of squared gradients = scale)}\\ w& \leftarrow w-\eta \frac{m}{\sqrt{v}+\epsilon }& & \text{(step: direction from }m\text{, size tamed by }\sqrt{v}\text{)}\end{array}$$

${\beta }_1\approx 0.9$ and ${\beta }_2\approx 0.95$ to $0.999$ control how long the “memory” lasts. Parameters with big, jumpy gradients get smaller steps; quiet ones get bigger steps. (The real algorithm also includes a small bias-correction term, omitted here.)

---

← 3. Loss and how wrong we are · Contents · 5. From generic networks to language →