TwiddleLab / Artificial Intelligence / Learning by Reward

Q-Learning

Lesson 4 of 10 · 7 min · Free, no account needed

What you will learn

• A Q-value is a note for one move in one square: how good that move turned out, counting what came after.
• Q-learning: after each move, nudge the note toward reward + best note in the new square.
• The learning rate says how far each nudge goes. Many episodes later, the best notes trace the best route.

Start the lesson

How the lesson goes

1. You've eaten at the same café twenty times. Nobody gave you ratings, but you know what's good. How?

   Now a critter in a meadow it cannot see. What should it write down after each move?

2. Step through the critter's first episode.

   It cannot see the meadow. After each move it updates one note: the triangle pointing the way it went.

3. Now do one update yourself.

   The target is what this move just proved it is worth. The note moves halfway there.

4. Let it learn for many episodes.

   Watch the returns and the arrows of its best notes.

5. This is Q-learning.

   The Q stands for quality: the quality of a move in a square.

6. Your turn in the dark: follow the notes to the food.

   The map is hidden, but a trained critter's notes are not.

7. Put one step of Q-learning in order.

   The loop runs once per move, over and over.

   The update needs two things the move produces: the reward and the new square. So the order is look, pick a move, get reward and new square, find the best note there, then nudge the old note.

8. In 2015 a program from DeepMind learned 49 arcade games this way.

   It saw only the screen pixels and the score. Label the parts.

   The screen pixels are the state, the joystick moves are the actions, the change in score is the reward, and the program is the agent. Its notes were too many to keep in a table, so a neural network estimated them: a deep Q-network.

9. What is the new note?

   The target is -1 + 6 = 5. The old note is 2, so the gap is 3. Half the gap is 1.5, and 2 + 1.5 = 3.5.

10. Test the expert: drop it anywhere.

    This critter practised 500 episodes. Tap a square to drop it there.

Previous lesson: Value: How Good Is This Square? · Next lesson: Explore or Exploit? · Part of the course: Learning by Reward

Lessons are interactive and need JavaScript. Turn it on to play them.