TwiddleLab / Artificial Intelligence / Learning by Reward

Learning to Play Games

Lesson 8 of 10 · 7 min · Free, no account needed

What you will learn

• Self-play: an agent learns a game by playing against a copy of itself.
• After each game, positions the winner faced score a little higher, the loser's a little lower.
• In big games a value network guesses how good a position is. AlphaGo Zero learned Go this way.

Start the lesson

How the lesson goes

1. In 2016 a program beat one of the world's best Go players. Its successor learned only by playing itself.

   How do you learn from a partner who knows as little as you? Try it on a small game.

2. Play the stick game against a bot that moves at random.

   Take 1, 2 or 3. Whoever takes the last stick wins.

3. Now let the bot play against itself.

   Each bar is the bot's guess that whoever must move from that pile will win. Watch them change.

4. This is self-play.

   The table of values is what the bot learned. Nobody told it the rule of 4.

5. Be the learning rule for one game.

   Move each touched bar toward the result: 1 for the winner's piles, 0 for the loser's.

6. Beat the trained bot. You can see its values.

   It always leaves you a pile it rates as losing for you. Turn that around.

7. Put the self-play loop together in Python.

   play_one_game returns which piles the winner faced and which the loser faced.

   Each game: play it, then walk through the winner's piles and push them toward 1, then the loser's piles toward 0. The outer loop repeats this 1000 times.

8. From which piles does the player to move lose?

   Assume the opponent never makes a mistake.

   From a multiple of 4, whatever you take (1 to 3), the opponent can take enough to make the pair add up to 4 and leave the next multiple of 4, all the way down to 0.

9. How did AlphaGo Zero (2017) learn to play Go?

   AlphaGo Zero got only the rules. It played itself millions of times, and a neural network learned to rate positions and suggest moves from those games. It then beat the earlier AlphaGo 100 games to 0.

10. Tournament: does each version beat the one before?

    Pick two versions of the bot and let them play 100 games.

Previous lesson: Reward Hacking · Next lesson: RL in Chatbots · Part of the course: Learning by Reward

Lessons are interactive and need JavaScript. Turn it on to play them.