Multiply · add · squash · repeat
What a neural network actually is
Not a brain, and not a database of answers. A neural network is a stack of very simple arithmetic, repeated enough times that the combination can describe complicated things. Start with one neuron, then put a few of them side by side and watch what becomes possible.
Part one
A neuron is a weighted opinion
A neuron takes some numbers in, multiplies each by a weight, adds them up along with a bias, and squashes the total into a 0–1 answer. That is the whole thing. The weights say how much each clue matters and in which direction; the bias says how easily convinced the neuron is. Drag anything below — the arithmetic is live.
Is this fruit ripe?
all three clues count, and it takes two of them to tip
The arithmetic
- Colour+1.44
- Softness+1.05
- Smell+0.96
- bias-2.40
total = 1.05
the squash: any total in, 0–1 out
ripe
Notice what the squash buys you. Without it, a neuron is just a weighted sum — and stacking weighted sums on top of each other collapses back into a single weighted sum, no matter how many layers you use. The squash is the bend. It is the reason a stack of layers can do more than one layer.
Part two
A layer is a committee
One neuron can only ever split the world with a straight cut: everything on one side is a yes, everything on the other a no. Put several neurons side by side, each drawing its own cut, and let a final neuron weigh up what they all said — now the boundary can bend, close into a loop, or double back on itself.
Below is a real network learning to separate two colours of dot, from scratch, in your browser. The wash of colour is its answer for every point on the square, not just the dots it was shown.
6 hidden neurons · 19 weights and biases
epoch 0 / 1400
So what is the network, really?
It is a function: numbers in, numbers out, with a long list of weights and biases sitting in the middle deciding what it does. Nothing in the structure knows about fruit, or dots, or language. The structure only provides enough flexibility to express a great many different functions; the weights pick out which one.
The demos here have between three and 25 numbers to set. A large language model has hundreds of billions, and its inputs are pieces of text turned into long lists of numbers rather than two coordinates. The layers are arranged more cleverly too — attention layers let each position in the text read from every other position rather than only from a fixed grid. But the primitive underneath is the one on this page: multiply by a weight, add a bias, squash, pass it on.
The remaining question is where the weights come from. Nobody sets them by hand. They start random and get nudged, over and over, by a process that works out which direction each one should move — that is backpropagation, and it has its own page.