Backprop: Blame Flows Backwards

Lesson 3 of 10 · 7 min · Free, no account needed

What you will learn

Backpropagation starts at the error and sends blame backwards, layer by layer.
Blame of a weight = blame at its end × the signal on its wire. That blame is the weight's gradient.
A wire carrying 0, or a neuron whose ReLU is off, gets no blame. One training step moves every weight by −lr × its blame.

The network guessed 9. The answer was 5. Which of its knobs deserve the blame?
Start with the three wires into the guess.
Nudge each last wire by +1. Which one moves the guess most?
Tap a wire into the guess. Its weight goes up by 1 for a moment.
Now send the blame all the way back.
Press Step to move the blame one layer back at a time.
This is backpropagation.
Every weight got its blame from one pass backwards.
Tap every wire that gets no blame at all.
The blame numbers are hidden. Use the signals.
The wire from h3 carries 0, so its blame is 4 × 0 = 0. h3's ReLU is off, so no blame flows back through it to the two wires that feed it.
One training step: land the new guess between 4.5 and 5.5.
Every weight moves against its blame, by step size lr × blame.
With lr 0.015 every weight moves by 0.015 × its blame, and the new guess is 4.66. Bigger steps overshoot.
Put one training step in order.
The loop repeats: after step 4 it starts again at 1.
Code: why not just nudge each weight and watch the error?
Count the runs both ways, like counting loop steps in code.
A new network guessed 2. The answer was 6. Which weight gets the biggest blame?
Each weight's blame is −4 × its signal: −12, −4, 0 and −8. The biggest in size is −12, on the wire carrying 3. Negative blame means: turn this weight up.
Train the network with your step size.
Press Train one step five times and watch the guess.

Lessons are interactive and need JavaScript. Turn it on to play them.