TwiddleLab / Artificial Intelligence / Inside Deep Learning

Project: Debug a Broken Network

Lesson 10 of 10 · 8 min · Free, no account needed

What you will learn

• Debug in order: read the curves, check the gradient bars, change one knob, train again and compare.
• Exploding loss: lower the learning rate. Memorizing: regularization or early stopping. Flat from the start: look for vanishing gradients.
• Common code bugs: a learning rate far too big, and data that is never shuffled.

Start the lesson

How the lesson goes

1. Three networks, three bug reports from your team. Fix them before the demo.

   Start with a guess about net 3, the one that never learns.

2. Match each bug report to its dashboard.

3. Net 3 is flat. Look at its gradient bars.

   Tap a bar to see its value, then Check.

   Layer 8 gets 0.5, layer 7 0.125, layer 6 0.03, layer 5 0.008. The early layers get almost no blame: a vanishing gradient.

4. A debugging checklist.

   Engineers who train networks follow some version of this every day.

5. Fix net 1: its loss explodes.

   Change only the learning rate.

6. Fix net 2: it memorizes.

   Use the weight penalty, dropout, or both. Too much and it stops learning.

7. Fix net 3: rebuild its layers so every layer learns.

   Switch layers between sigmoid and ReLU, and pick the starting weights.

8. Write down your checklist, in order.

9. Code: a teammate's training script has two bugs. Find and fix them.

   Tap a line to inspect it.

10. A new run. Which one knob fixes it?

    Training loss keeps falling while validation turns up: overfitting. Dropout (or a weight penalty, or stopping earlier) fights it.

11. Demo day: run all three fixed networks.

Previous lesson: Reading Training Curves · Part of the course: Inside Deep Learning

Lessons are interactive and need JavaScript. Turn it on to play them.