1. The artificial neuron

The “neuron” in a neural network is a loose, very lossy analogy to a biological one. Mathematically it’s almost embarrassingly simple: multiply each input by a weight, add them up, add a bias, then squash the result.

$y = f (w_{1} x_{1} + w_{2} x_{2} + \dots + w_{n} x_{n} + b)$

Written compactly with the sum symbol (see Symbol cheat sheet):

$y = f (\sum_{i = 1}^{n} w_{i} x_{i} + b)$

And in vector notation, which is what you’ll see everywhere:

$y = f (w \cdot x + b)$

Breaking down the formula

Piece Name What it is
$x_{1} \dots x_{n}$ inputs The numbers coming in
$w_{1} \dots w_{n}$ weights Learned numbers saying how much each input matters
$b$ bias A learned offset (lets the neuron fire even when inputs are zero)
$\sum_{i = 1}^{n} w_{i} x_{i}$ weighted sum Multiply each input by its weight, then add all of them
$f$ activation function A non-linear squashing or gating function
$y$ output The single number the neuron produces

That dot product is the core operation of the whole field. Everything else is organization around it.

Piece	Name	What it is
$x_{1} \dots x_{n}$	inputs	The numbers coming in
$w_{1} \dots w_{n}$	weights	Learned numbers saying how much each input matters
$b$	bias	A learned offset (lets the neuron fire even when inputs are zero)
$\sum_{i = 1}^{n} w_{i} x_{i}$	weighted sum	Multiply each input by its weight, then add all of them
$f$	activation function	A non-linear squashing or gating function
$y$	output	The single number the neuron produces

flowchart LR
    x1["x₁"] -->|"× w₁"| S(("Σ + b"))
    x2["x₂"] -->|"× w₂"| S
    x3["x₃"] -->|"× w₃"| S
    S --> F["f( ) activation"]
    F --> Y["output y"]

Worked example with real numbers

Inputs $x = (2, - 1, 3)$ , weights $w = (0.5, 1, 0.2)$ , bias $b = 0.1$ , activation = ReLU.
$weighted sum add bias y = (0.5) (2) + (1) (- 1) + (0.2) (3) = 1 - 1 + 0.6 = 0.6 = 0.6 + 0.1 = 0.7 = ReLU (0.7) = max (0, 0.7) = 0.7$
If the pre-activation had been negative (say $- 0.5$ ), ReLU would output $0$ . The neuron “stays off.”

Common activation functions

Name	Formula	Notes
ReLU	$max (0, x)$	Dead simple, was the default for years. “If negative, output 0, else pass through.”
GELU	$x \cdot Φ (x)$	A smooth ReLU-ish curve. Used in GPT-2/BERT era. $Φ$ is defined in the Symbol cheat sheet.
SiLU / Swish	$x \cdot σ (x)$ , where $σ (x) = \frac{1}{1 + e ^{- x}}$	Common in modern models. $σ$ (the sigmoid) squashes any number into the range 0 to 1.
SwiGLU	$(SiLU (x W_{gate}) ⊙ x W_{up}) W_{down}$	A gated variant: one path decides “how much to let through” for the other. What most modern LLMs use in their MLPs (see 8b. The MLP).

← The big picture · Contents · 2. Layers and why non-linearity matters →

Quartz 5

Explorer

1. The artificial neuron

Common activation functions

Backlinks