Symbol cheat sheet

"What's the sideways M?"

That’s Σ, the capital Greek letter sigma. It just means “add up a bunch of things.” It looks like an M lying on its side, but it’s a sum.

${\sum }_{i=1}^3{x}_i=x_1+x_2+x_3$

Read it as: “for i going from 1 to 3, take x sub i, and add them all up.” The number under the Σ is where you start, the number on top is where you stop. If $x=(2,5,7)$, the sum is $2+5+7=14$.

It’s the same idea as a for loop with a running total:

total = 0
for i in range(1, 4):
    total += x[i]

Every symbol used in this note, in one place:

Symbol	Say it as	What it means	Tiny example
$\sum$	”sum” (capital sigma)	Add up a list of terms	${\sum }_{i=1}^{3}i=1+2+3=6$
$\sigma$	”sigma” (lowercase)	The sigmoid function, squashes any number into 0 to 1. Different from Σ!	$\sigma (0)=0.5$
$\partial$	“partial"	"A tiny change in…”. Used in derivatives	$\frac{\partial L}{\partial w}$ = how much the loss moves when $w$ moves a hair
$\theta$	”theta”	Shorthand for all the model’s parameters (every weight and bias)	$p_{\theta }$ = probabilities from the model with parameters $\theta$
$\eta$	“eta”	The learning rate, i.e. the step size	$\eta =0.01$
$\epsilon$	”epsilon”	A tiny number added to avoid dividing by zero	$\epsilon =10^{-6}$
$\mathbf{x}$	”x vector”	A bold letter means a list of numbers (vector)	$\mathbf{x}=(2,-1,3)$
$W$	“W”	A capital letter usually means a grid of numbers (matrix)	A $3\times 4$ grid
$\mathbf{x}\cdot \mathbf{w}$	”dot product”	Multiply matching entries, then add them all up	$(1,2)\cdot (3,4)=3+8=11$
$\odot$	”elementwise times”	Multiply two lists entry-by-entry (no adding)	$(1,2)\odot (3,4)=(3,8)$
$K^{\top }$	“K transpose”	Flip a grid so rows become columns	A $5\times 8$ grid becomes $8\times 5$
$\in$	“is in”	Belongs to a set	$x\in \mathbb{R}$ = “x is a real number"
${\mathbb{R}}^d$	"R to the d”	A list of $d$ real numbers	${\mathbb{R}}^{4096}$ = a 4096-long vector
$x_i$	”x sub i”	The i-th entry of $x$ (the little number is just a label)	$x_2$ = second entry
$x_{<t}$	”x before t”	Everything that came before position $t$	For $t=4$: $x_1,x_2,x_3$
$\leftarrow$	“gets” / “becomes”	Overwrite the left side with the right side (assignment)	$w\leftarrow w-1$
$\approx$	“approximately”	Close to, not exactly equal	$\pi \approx 3.14$
$\sqrt{x}$	”square root”	The number that squared gives $x$	$\sqrt{16}=4$
$e^x$ or $\exp (x)$	“e to the x”	Exponential. $e\approx 2.718$. Always positive, grows fast	$\exp (0)=1$, $\exp (1)\approx 2.72$
$\log (x)$	“log”	Natural log, the inverse of $\exp$. Negative for $x<1$	$\log (1)=0$, $\log (0.5)\approx -0.69$
$\infty$	“infinity”	Bigger than any number	$-\infty$ is used to “turn off” things
$\Phi (x)$	”capital phi of x”	Probability that a standard bell-curve random number is below $x$ (used in GELU)	$\Phi (0)=0.5$

Two more reading tips

• $p(a\mid b)$ is read “probability of $a$ given $b$.” The tall bar means “given” or “knowing that.”
• A fraction like $\frac{\partial L}{\partial w}$ is just one symbol-group. Don’t try to cancel the $\partial$‘s like normal fractions.

---

Contents · The big picture →