TwiddleLab / Artificial Intelligence / Learning by Reward

Value: How Good Is This Square?

Lesson 3 of 10 · 7 min · Free, no account needed

What you will learn

• The value of a square is the best total the agent can still get from there.
• The value rule: value = reward of the best move + value of the square it lands on.
• Values spread back from the goal. Stepping to the best neighbour every time is the best route.

Start the lesson

How the lesson goes

1. A chess player glances at a board and says: I'm winning.

   A position can be worth something before the game ends. So can a square. Which one is worth more to the critter?

2. Give every square its value, starting at the food.

   Tap a square next to one that already has a value. It works out its own.

3. Now let every square use the rule at once.

   Press Sweep once a few times and watch the values spread.

4. What you built has a name.

   Richard Bellman wrote this rule down in the 1950s. It is at the heart of learning by reward.

5. Climb the hill: walk from the start to the food.

   Tap squares one move apart. Can your total match the start's value?

6. Move the food so the start is worth exactly 7.

   Tap a square to move the food there. The values follow.

7. One number on this map is wrong. Find it.

   Tap a square to test it against its neighbours.

8. Same trick, new problem: how many ways up the stairs?

   You climb 1 or 2 steps at a time. Count the ways from the top down.

9. What is the value of the square with the question mark?

   Only its neighbours are shown.

   The best neighbour is worth 7. Moving there costs -1. So the value is -1 + 7 = 6. The rock above is not a move.

10. Plant some thorns and watch the hill reshape.

    Then flip between Before and After.

Previous lesson: Trial and Error · Next lesson: Q-Learning · Part of the course: Learning by Reward

Lessons are interactive and need JavaScript. Turn it on to play them.