The Chain Rule in a Network

Lesson 4 of 10 · 7 min · Free, no account needed

What you will learn

A network is a chain of small steps. Each step has a local rate: how fast its output moves when its input moves.
The chain rule: the rate of the whole chain is the product of the local rates.
Backprop multiplies the rates from the error backwards. A ReLU that is off has rate 0 and stops the blame.

This knob sits four steps away from the error. How much does turning it change the error?
w feeds z, z feeds a, a feeds y, and y gives the error.
Gears: B turns 3 times as fast as A, C turns 2 times as fast as B.
Turn A so that C makes exactly 12 turns.
A network is a chain too. Nudge w and follow the wiggle.
Watch how much each value moves.
Build the total rate: multiply from the error backwards.
Tap the rates on the links, starting next to the error.
This is the chain rule.
It is the same rule as the gears, used on every weight of a network.
Now w = −1. Which link stops all blame from reaching w?
With z = −2 the ReLU outputs 0 and stays 0 for small nudges, so its rate is 0. The product 2 × 0 × 3 × (−4) is 0: w gets no blame.
One rate in this chain is wrong. Find it and fix it.
The chain is the same as before: w = 1, so y = 6.
Math: (3x + 1)² is a chain of two gears too.
Find the x where the total rate is 24.
The four local rates are 0.5, 3, 1 and 4. What is the total rate from w to the error?
The chain rule multiplies the local rates: 0.5 × 3 × 1 × 4 = 6.
Make your own chain with total rate exactly 24.
Drop a rate onto each of the four links. Fewer moves, more stars.

Lessons are interactive and need JavaScript. Turn it on to play them.