Fighting Overfitting

Lesson 7 of 10 · 7 min · Free, no account needed

What you will learn

A big network can memorize its training data: zero training error, big error on new data.
Regularization makes memorizing harder. Weight decay adds λ × (sum of squared weights) to the loss.
Dropout switches off random neurons on every step. Too much regularization underfits: find the λ where new-data error is lowest.

A huge network memorizes every training photo. You can't always get more data. Can you make memorizing harder?
First a bet: two curves trained on the same 10 dots.
Train a big network on 10 dots, then test it.
20 neurons, 10 training dots. Press Train, then Show new data.
Charge the network for big weights. Get the new-data error below 0.45.
λ (lambda) is the price per unit of squared weight.
Another trick: switch off random neurons while training.
Pick a dropout rate. Get the new-data error below 0.42.
Both tricks are regularization.
They trade a little training error for a lot less error on new data.
Which ideas close the gap between training and new data?
More data and flipped or rotated copies (data augmentation) give more patterns to learn. Dropout and weight decay make memorizing harder. More epochs and more neurons give more room to memorize.
Find the best penalty: the λ with the lowest error on new dots.
Every λ you try leaves a dot on the small chart. Then press Check.
Error on new dots: 0.78 at λ 0, 0.42 at 0.3, 0.38 at 1, 0.64 at 3. λ = 1 is the bottom of the valley.
Math: a penalty moves the minimum.
The error alone, (w − 4)², is smallest at w = 4. Add the penalty 1 × w² and find the new bottom.
Which of these fits is overfitting?
Fit B passes through every training dot (error 0) but misses the new dots (0.78). Fit A is too flat: it underfits. Fit C is the balanced one.
Before and after: your fix.
Switch between the network without a penalty and with your λ.

Lessons are interactive and need JavaScript. Turn it on to play them.