Reward Hacking

Lesson 7 of 10 · 7 min · Free, no account needed

What you will learn

Reward hacking: an agent finds a way to score high that nobody intended.
It is not cheating on purpose. The agent does exactly what the numbers reward.
Reward design: reward the result you want, then watch what the trained agent really does, and fix.

You reward a racing boat for every coin it collects. It never finishes the race. Why?
Train the boat on +1 per coin.
Fix the reward so the boat races straight to the finish.
Add a bonus for finishing, a penalty for every move, or both. The list shows what each plan would earn.
This has a name: reward hacking.
It happened in 2016 to a boat-racing agent at OpenAI in the game CoastRunners: it circled a lagoon hitting targets and never finished.
A robot vacuum needs a reward. Find one it can't game.
Tap a rule to train with it and see what the robot learns.
Hack hunt: three real agents, three loopholes.
Tap the part of each picture where the agent games its reward.
Patch each loophole.
A good fix measures the real goal more directly: time spent paused, an actual touch, the block's real position. Making the old reward bigger only makes the old loophole pay more.
In code, tests are a reward too.
Program B passes a weak test without reversing anything. Choose tests that catch it.
A warehouse robot gets +1 every time it scans a package. What loophole might it find?
The reward counts scans, not packages delivered. Scanning one package over and over is the cheapest way to collect +1s. A better reward: +1 per package that arrives at the right shelf.
Race day with your reward.
Your reward from earlier, on a new race.

Lessons are interactive and need JavaScript. Turn it on to play them.