TwiddleLab / Artificial Intelligence / Learning by Reward

Discounting the Future

Lesson 6 of 10 · 7 min · Free, no account needed

What you will learn

• The discount factor γ says how much a reward one move later still counts.
• Return = r1 + γ r2 + γ² r3 + ... : later rewards shrink by γ each move.
• Small γ: grab what is near. γ close to 1: walk further for something bigger.

Start the lesson

How the lesson goes

1. One cookie now or two cookies tomorrow? Machines have to answer this too.

2. Find the smallest γ that sends the critter to the cake.

   Each move further away, a reward counts γ times less. Slide γ and watch both numbers.

   The cake arrives on move 6, so it counts 10 × γ⁵. At γ = 0.70 that is 1.68, less than the berry's 2. At 0.75 it is 2.37, more than 2.

3. Two critters, two values of γ. Where do they disagree?

   Tap every square where the arrows on the two maps point different ways.

   At γ = 0.5 the cake fades fast, so squares near the berry head for it. At γ = 0.9 the cake is still worth more almost everywhere, so those squares head for the cake.

4. The number you were sliding has a name.

   It is the Greek letter gamma. People and animals discount the future too.

5. What is this reward worth right now?

   Move 1 counts fully, move 2 times γ, move 3 times γ², move 4 times γ³. So 20 × 0.5³ = 20 × 0.125 = 2.5.

6. Get the critter to the cake without crossing the brambles.

   Brambles cost -1 each. The long way round has none.

7. With γ = 0.9, after how many moves is a reward worth less than half?

   Each bar is 0.9 to the power n. Tap the first bar below the line.

   Multiplying by 0.9 again and again shrinks a number fast: 0.9⁷ ≈ 0.48. This shrinking by the same factor each step is exponential decay, the same pattern as a cooling cup or a fading echo.

8. Set γ so the critter goes for the cake: not the berry, not the feast.

   The cake beats the berry once 10γ² > 2 (γ above about 0.45) and beats the feast while 10γ² > 30γ⁸ (γ below about 0.83). Anything from 0.45 to 0.80 on this slider works.

9. Race: three critters, three values of γ.

   Who collects the most? Pick one, then press Race.

Previous lesson: Explore or Exploit? · Next lesson: Reward Hacking · Part of the course: Learning by Reward

Lessons are interactive and need JavaScript. Turn it on to play them.