TwiddleLab / Artificial Intelligence / Learning From Data
Accuracy Can Lie
What you will learn
- When one kind is rare, a model can score high accuracy by always guessing the common kind.
- A confusion matrix splits answers into caught, missed, false alarms and correctly cleared.
- A false positive is a false alarm, a false negative is a miss. Which is worse depends on the job.
How the lesson goes
A model screens 100 critters for glowpox and is 95% accurate. Would you trust it to find the sick ones?
5 of the 100 are really sick. Drag the marker: how many of them does it find? Then Check.
Run the 95% model.
Press Run and watch it screen all 100 critters.
Sort 12 screened critters into four boxes.
Each card says what's real and what the model said. Drag it into its box, or tap the card and then the box. Then Check.
These four boxes have names.
Tap to reveal them.
The 2 by 2 grid is a confusion matrix: what's real against what the model says.
A false positive is a false alarm: flagged, but healthy. A false negative is a miss: sick, but not flagged.
When one kind is rare, that's class imbalance. Accuracy can look great while every rare case is missed.
Which mistake is worse? It depends on the job.
In each row, tap the mistake that costs more. Then Check.
Beat the lazy model.
You're the screener now. You have 8 flags. Flag the critters you think are sick and find at least 4 of the 5.
Math: 1 in every 1000 card payments is fraud. A model says "not fraud" every time.
Out of 1000 payments, how many does it get right?
Two screeners checked the same 200 new critters. Which one goes to the hospital?
Open both matrices, then tap a screener and Check.
Write your screener's report card.
Your 8 flags made these four numbers. Drag each into its box of the matrix. Then Check.
Previous lesson: Train, Test, Don't Peek · Next lesson: Where to Draw the Line: Precision and Recall · Part of the course: Learning From Data
Lessons are interactive and need JavaScript. Turn it on to play them.
