TwiddleLab / Artificial Intelligence / Learning From Data
Biased Data, Biased Machine
What you will learn
- Data bias: if a group is rare in the training data, the model learns little about it and fails it more often.
- Sampling bias often comes from convenience: whoever is easiest to reach fills the dataset.
- Check per-group accuracy. An overall score can look fine while one group is failed.
How the lesson goes
A face filter works great on your friends but glitches on your cousin. Same app, same phone. Why?
Same idea with critters. An app learned mostly from forest critters. Drag the marker: how well will it do on mountain critters? Then Check.
Collect the training data yourself.
Press Start. Critters walk past your net, one at a time. Press Catch while one is inside the net. 12 places.
Train on your set. Test on both groups.
Press Train and test. 20 new forest critters and 20 new mountain critters take the exam.
What if the data had more mountain critters?
Slide how many of the 20 training critters come from the mountains. Watch both groups.
Names for what just happened.
Tap to reveal them.
Data bias: the training data doesn't represent everyone the model will meet.
Sampling bias is one cause: some groups are easier to collect, so they fill the data.
Per-group accuracy checks each group separately. An overall score can hide a group the model fails.
Field trip budget.
You have 30 coins. A forest critter costs 1, a trip to the mountains costs 3. Build a training set where both groups score 85% or more.
The park must pick one of two apps.
Open both apps by group. Then tap the one the park should use, and Check.
Math: where you ask decides what you hear.
The town asks 1000 people which park to build. Drag the clipboard to a place to survey. Try a few, then Check with the fairest one.
Where will this photo app fail?
Tap the group it will most likely get wrong, then Check.
Make a fair data card for the critter app.
Drag each sticker to its line. Your field trip result is filled in. Then Check.
Previous lesson: Finding Groups: Clustering · Next lesson: How Your Feed Picks: Recommendations · Part of the course: Learning From Data
Lessons are interactive and need JavaScript. Turn it on to play them.
