Reading Training Curves

Lesson 9 of 10 · 7 min · Free, no account needed

What you will learn

A learning curve shows the loss by epoch, on training data and on validation data.
Overfitting: training keeps falling while validation turns up. Early stopping keeps the model from validation's lowest point.
Underfitting: both stay high. Too big a learning rate: the curves jump around. The slope of a curve is the speed of learning.

Your model trained all night. Two lines on a chart. Good news or bad?
You only see the first 15 epochs. Sketch the rest of the orange line.
Here is the whole night. Where should training have stopped?
Validation loss is lowest around epoch 20. After that the model gets better on training data and worse on new data.
Four models, four stories. Label each chart.
Learning curves, early stopping, underfitting.
Every training run produces these two lines. Reading them is the first debugging skill.
Stop the training live, when validation is lowest.
Press Start, watch the orange line, press Stop. Within 3 epochs wins.
Which learning rate made the grey curve?
Try learning rates until your purple curve matches.
Prescribe a fix for each broken model.
Math: the slope of the curve is the speed of learning.
Find a point where learning has almost stopped.
Which model is overfitting?
In model C training loss keeps falling while validation loss turns up after epoch 20. A is healthy, B underfits.
Speed round: six charts, six seconds each.
Tap the diagnosis before time runs out.

Lessons are interactive and need JavaScript. Turn it on to play them.