Explore or Exploit?

Lesson 5 of 10 · 7 min · Free, no account needed

What you will learn

Exploit: use the best option you know. Explore: try something else to learn more.
A purely greedy agent can get stuck after one unlucky try.
Epsilon-greedy: with a small chance ε pick at random, otherwise pick the best so far.

Your favorite pizza place, or the new one that just opened?
The critter has the same problem with three berry machines.
20 pulls. Collect as many berries as you can.
Every pull shows what that machine gave. Which one is best?
Watch a greedy agent play.
It tries each machine once, then always pulls the one with the best average so far.
Add a little randomness: reach 6.5 berries per pull.
Set ε, the chance that a pull goes to a random machine, and press Run.
This trade-off has a name: explore or exploit.
Every agent that learns by trying faces it.
With ε = 0.2, about how many of 100 pulls are random?
Each pull is random with chance 0.2, so out of 100 pulls about 0.2 × 100 = 20 are random. The exact number changes from run to run.
What if the world changes?
Pick who you think does better after the change, then run.
Machine C's average over these 10 pulls?
This is how agents estimate a machine's payout: add up and divide.
5 pulls gave 10 and 5 gave nothing: 50 berries in 10 pulls, 50 / 10 = 5. In maths this is an estimate of the expected value: C's true chance is 50%, so it pays 0.5 × 10 = 5 per pull on average.
A music app plays your favorite songs 9 times in 10, and something new 1 time in 10. What is it doing?
Most of the time it exploits what it knows you like, and with chance 0.1 it explores something new, to learn more about your taste. That is epsilon-greedy with ε = 0.1.
Make your strategy card.
Choose your ε and see how it stacks up.

Lessons are interactive and need JavaScript. Turn it on to play them.