TwiddleLab / Artificial Intelligence / How a Neural Network Thinks
Why Networks Get Fooled
What you will learn
- A model that learns its training data too closely, noise and all, is overfitting: perfect on old data, bad on new.
- A model too simple for the pattern is underfitting: bad on both.
- Hold back a validation set. Its error is the honest score.
- More flexibility never raises training error. Validation error makes a U: aim for its bottom.
How the lesson goes
An AI scored 100% on every practice question, then failed badly on real ones. How?
A plant grew for 8 days. Two models learned its growth, and B passes through every point. Which one would you trust for day 9? Tap one, then Check.
Start simple: one straight line through the growth.
Drag both ends of the line. The card is the training error from lesson 4: get it under 1, then Check.
Now let the line bend.
Turn up the flexibility until the training error hits 0, then Check.
Is a perfect score a good model?
Press New data: measurements of the same plant the model never trained on. Then press Sweep to try every flexibility.
This trap has a name.
Tap to reveal.
Learning the training data so closely that the model learns its noise too is overfitting: perfect on old data, bad on new.
A model too stiff to follow the pattern is underfitting: bad on both.
Data you hold back from training is the validation set. Its error is the honest score, and its lowest point is the sweet spot.
Game: Sweet spot.
The machine always uses flexibility 4. On each dataset, pick your flexibility by eye, then Test on new data. Beat the machine on all three.
Too simple, just right or too flexible?
Each card is a model with its training and new-data error. Put every card in its zone, then Check.
In math, a polynomial of degree 9 can pass through any 10 points.
Break it: nudge the orange point a little and watch the whole curve. Then fix it: lower the degree until the curve stays calm, then Check.
Which model is overfitting?
Each chart shows the training error and the new-data error while a model trains. Tap the overfitted one, then Check.
Day 9: your model against your bet.
Green is your plant model from the game. Dotted violet is the model you bet on. Press Measure day 9.
Previous lesson: Networks Solve XOR · Next lesson: Project: Train Your Own · Part of the course: How a Neural Network Thinks
Lessons are interactive and need JavaScript. Turn it on to play them.
