TwiddleLab / Artificial Intelligence / Learning From Data

Train, Test, Don't Peek

Lesson 4 of 11 · 7 min · Free, no account needed

What you will learn

• A model learns from the training set and is judged on a test set it has never seen.
• Use a validation set to choose settings. Open the test once, at the end.
• Overfitting: memorizing the training examples, odd ones included, and doing worse on new ones.
• If test examples leak into training or steer your choices, the test score stops telling the truth.

Start the lesson

How the lesson goes

1. A classmate memorized last night's exam answers. Did they learn the subject?

   Same question for a tree. It got all 30 critters it studied right. Drag the marker: what will it score on 30 new critters? Then Check.

2. Give it the two exams.

   Drag the time slider or press Play: first the critters it studied, then 30 new ones.

3. How many questions should the tree be allowed to ask?

   Slide the limit and watch both scores.

4. Cheat on the exam.

   The exam critters are locked in a vault. Drag the exam paper onto the study desk and let the tree study it. Then release the tree into the world.

5. Every machine learning team uses these words.

   Tap to reveal them.

   The training set is what the model learns from: the filled dots.

   A validation set is a second batch you use to choose things, like how many questions to allow.

   The test set stays in the vault until the very end. It's the real exam, used once.

   Split first, then never let test critters into training. If you do, the test stops telling the truth.

6. One look in the vault.

   Choose how many questions with the validation score, then open the vault once. Win with 80% or more on the test.

7. Split 10 critter photos: 8 to study, 2 to test.

   Careful: some critters were photographed twice. Drag each photo into a pile, or tap it and then a pile. Then Check.

8. Code: a split is a slice.

   After shuffling, a program cuts the list in one place. Drag the scissors so 20% of the critters go to the test. Then Check.

9. A team describes how they tested. Tap every habit that leaks the test.

   Tap the notes, then Check.

10. Release both of your models.

    Your cheater from step 4 and your honest model from step 6. Press Release both.

Previous lesson: Twenty Questions: Decision Trees · Next lesson: Accuracy Can Lie · Part of the course: Learning From Data

Lessons are interactive and need JavaScript. Turn it on to play them.