TwiddleLab / Artificial Intelligence / AI and Us: Using AI Well
Bias: Whose Data?
What you will learn
- Historical bias: a model learns from past decisions. If the past was unfair, the model learns that and repeats it.
- Deleting the sensitive field isn't enough: a proxy, like a postcode, can carry the same pattern.
- Check results by group. The four-fifths rule flags a group whose success rate is under 0.8 of the best group's rate.
- The fix is in the data: re-review unfair labels, then retrain.
How the lesson goes
Same grades. Same project. Same letter. An AI accepted one and rejected the other.
A summer coding camp screens applicants with a model. Make a bet: tap the field you think split them. Then Check.
Where did that come from? Train the model and watch.
It learns from 20 past decisions made by people. Green got in, orange didn't.
Fix the data. Re-review the unfair River rejections.
Hill applicants got in from a grade of 70. Tap each River rejection that would have passed at Hill. The model retrains after every flip.
The model only knows what the past decisions told it. When those decisions asked more of River applicants, the model learned to do the same. Fixing the labels fixes what it learns.
What you just fixed has a name.
Historical bias: the training data is a record of past decisions, and the model repeats their pattern, fair or not.
Each past decision is a label. Unfair labels teach an unfair model, even if nobody wrote an unfair rule.
A proxy is a field that stands in for another one. You'll meet one next.
The camp deletes the School field. Which other details could give the school away?
Drag each detail into a lane.
A model doesn't need the word School if another field carries the same pattern. That field is a proxy.
School is gone and the model retrained, but Ben is still rejected. Which column gives the school away?
Tap a column header, then Check.
Every Hill applicant lives in 4010 and every River applicant in 4021. The model learned "4021 needs more", which is the old school pattern under a new name.
Math: the four-fifths check. How many River applicants must get in?
Hill: 8 of 12 accepted, 67%. A group's rate divided by the best rate should be at least 0.8. Slide to the smallest number of River acceptances (of 8) that passes, then Check.
The four-fifths rule comes from US hiring guidelines. It is a warning sign, not proof: a ratio under 0.8 says "look closer", one above it doesn't prove the process is fair.
Two ways to fix the camp's model. Try both on Ben.
Then pick the fix that actually works and Check.
Hiding a field hides the word, not the pattern. Fixing the labels removes the pattern the model would learn.
A bank's model never sees anyone's background, but it does use home address. Can it still be biased?
If people from one background mostly live in certain areas, the address carries that pattern. The model can be unfair without ever seeing the background field.
Before and after. Run four new applicants through both models.
The old model learned from the original decisions, the fixed one from your re-reviewed data.
Previous lesson: Where AI Already Is · Next lesson: Hallucinations and Fact-Checking · Part of the course: AI and Us: Using AI Well
Lessons are interactive and need JavaScript. Turn it on to play them.
