TwiddleLab / Artificial Intelligence / Inside Deep Learning

Initialization and Exploding Gradients

Lesson 8 of 10 · 7 min · Free, no account needed

What you will learn

• Going back through a layer multiplies the blame by its local rate. Over many layers the rates multiply.
• Rates below 1 make the blame vanish, rates above 1 make it explode. 0.5 over 20 layers is about one millionth.
• Good initialization (for ReLU, He: spread √(2 ÷ inputs)) keeps each layer's rate near 1.

Start the lesson

How the lesson goes

1. Add more layers and the network should get smarter. Instead it stops learning at all. What went wrong?

   Here is a network with 20 layers.

2. Set the local rate so layer 1 gets a useful amount of blame.

   Every layer multiplies the blame by the same rate r on its way back.

3. The starting weights set the rate. Pick their spread.

   Keep the signal between 0.5 and 2 after 20 layers.

4. Vanishing and exploding gradients.

   Both come from multiplying many rates. Initialization is the first cure.

5. Each layer passes back 0.6. From which layer on is the blame under 1%?

   Tap a bar to see its value, then Check.

   0.6⁹ = 0.0101 is still just above 1%. 0.6¹⁰ = 0.006 is the first one below.

6. 10 layers, each halving the blame. How many times smaller is it at the end?

   Each layer halves it, so after 10 layers it is 0.5¹⁰ = 1 ÷ 2¹⁰ = 1 ÷ 1024.

7. Vanishing or exploding? Sort the symptoms.

8. Math: place 1.1⁵⁰ and 0.9⁵⁰ on the number line.

   Fifty layers, 10% more or 10% less each time.

   1.1⁵⁰ ≈ 117 and 0.9⁵⁰ ≈ 0.0052: exponential growth and decay, the same as in compound interest.

9. Build a 20-layer network where every layer learns.

   Pick the activation and the starting spread, then Train. Fewer tries, more stars.

Previous lesson: Fighting Overfitting · Next lesson: Reading Training Curves · Part of the course: Inside Deep Learning

Lessons are interactive and need JavaScript. Turn it on to play them.