TwiddleLab / Artificial Intelligence / Learning From Data

Project: Be the Data Scientist

Lesson 11 of 11 · 8 min · Free, no account needed

What you will learn

• A pipeline: data, split, model, evaluate, report.
• Data leakage makes scores too good to be true: drop columns known only after the answer.
• Choose on validation, test once, check groups, set the threshold for the job.
• A model card says what the model does, how it was tested and where it fails.

Start the lesson

How the lesson goes

1. A wildlife park wants an app that names the critters in visitors' photos. You're the data scientist. Where do you start?

   With data. Pick the dataset for your project, then Check.

2. Sort the columns before anything else.

   Drag each column head to feature, label, leak or junk. Then Check.

3. How much goes into the vault?

   Slide how much of the data you lock away for the final test.

4. kNN or a decision tree?

   Try both on the validation pile. Then tap the one you'll take forward, and Check.

5. You're running a pipeline.

   Tap to reveal its parts.

   A pipeline: data, split, model, evaluate, report. Every real project runs these steps.

   Data leakage: a column or a habit that lets the answer sneak into training. It makes scores look better than the truth.

   A model card reports what the model does, how it was tested and where it fails.

6. Classic: 85% on the sealed test.

   Tune your model on validation with - and +. Then open the vault. You need 85% or more on the test.

7. Evaluate by group.

   Press the button and watch your model's score for each group.

8. Challenge: a rare class, 3 in 100.

   Drag the line until recall is at least 0.8 and precision at least 0.5. Then Check.

9. Code: a pipeline is a short program.

   Put the lines in the order the program runs them. load, fit, check and report are made-up helpers.

10. A teammate's model scores 99%. Find out why.

    Tap a column to switch it off. Leave the model with only honest columns, then Check.

11. Your model card has a line "Where it fails". What should go there?

12. Fill in your model card.

    Tap each line. It fills in from what you did in this project.

Previous lesson: How Your Feed Picks: Recommendations · Part of the course: Learning From Data

Lessons are interactive and need JavaScript. Turn it on to play them.