TwiddleLab / Artificial Intelligence / Learning From Data
Where to Draw the Line: Precision and Recall
What you will learn
- A model gives each example a score. The threshold decides who gets flagged.
- Precision: of the flagged, how many were right. Recall: of the real cases, how many were caught.
- Moving the threshold trades misses for false alarms. Which mistake is worse depends on the job.
How the lesson goes
A smoke alarm that never goes off by mistake might also sleep through a real fire. Where would you set it?
Drag the violet line: the alarm beeps when smoke reaches it. Then Check.
A week in a real kitchen.
Press Play. Toast, steam, a barbecue and two real fires. Your line from step 1 decides when it beeps.
Glowpox again: now the model gives every critter a score.
Every critter right of the line gets flagged. Drag the line and watch the two meters.
At line 60, seven critters are flagged. Tap every false alarm.
Tap the flagged dots that are really healthy, then Check.
Two numbers tell you how good the line is.
Tap to reveal them.
The model's number for each critter is a score. The line you draw is the threshold.
Precision: of the critters flagged, how many are really sick? Here 4 of 7.
Recall: of the critters really sick, how many got flagged? Here 4 of 5.
Lower the threshold and recall goes up while precision usually goes down. Raise it, and the reverse.
Three houses, three alarms.
Each house has its own week of smoke. Drag each line so no fire is missed and it beeps for nothing at most once.
At line 60, the model flags 7 critters. 4 of them are really sick.
What's its precision, in percent? Round to a whole number.
Math: precision and recall are two fractions.
Build both from the counts at line 60. Drag the chips into the slots, then Check. (Conditional probability, from the Probability bridge.)
Which comes first for each job: precision or recall?
Drag each app to a side, or tap it and then a side. Then Check.
Factory setting against your three lines.
Every alarm left the factory set to 50. Press the button to swap in your lines from Three houses.
Previous lesson: Accuracy Can Lie · Next lesson: Scale Your Features · Part of the course: Learning From Data
Lessons are interactive and need JavaScript. Turn it on to play them.
