One neuron is useless alone. You stack many side by side to form a layer, and many layers to form a network.
A layer with 4 inputs and 3 neurons is just a matrix multiply:
Breaking down the formula
Piece Shape Meaning 4 numbers The inputs grid One row of weights per neuron 3 numbers Each neuron’s weighted sum, all computed at once 3 numbers One bias per neuron Applied to each of the 3 numbers separately 3 numbers The layer’s output
That’s why GPUs matter. A neural network is mostly giant matrix multiplications, and GPUs are built to do those in parallel.
Why the activation function is non-negotiable
If you remove , then stacking layers is pointless:
Two matrices multiplied together is just one matrix. Without non-linearity, a 100-layer network collapses into a single linear transformation and can only learn straight-line relationships. The non-linearity is what lets stacked layers represent complicated functions.
Universal approximation theorem: a network with enough neurons and a non-linearity can approximate basically any continuous function. This is a “this can work in principle” result. It says nothing about whether you can actually find the right weights, which is the training problem.
What "parameters" means
The weights and biases across the whole network are the parameters. When someone says “a 70B model,” they mean 70 billion of these numbers. Those numbers are the model. Nothing else is stored: no rules, no database, no facts in a table. Just numbers.
Try it
Watch a tiny network carve up a 2D plane as you add neurons and layers: TensorFlow Playground.
← 1. The artificial neuron · Contents · 3. Loss and how wrong we are →