Smarter Steps: Momentum and Adam

Lesson 6 of 10 · 7 min · Free, no account needed

What you will learn

In a narrow valley plain gradient descent zigzags: a step big enough for the floor is too big for the walls.
Momentum keeps a velocity: v = β × v + slope, then move by lr × v. Pushes that agree add up, back-and-forth ones cancel.
Adam gives each knob its own step size. Rules like these are called optimizers.

Rolling down a narrow valley, plain gradient descent zigzags from wall to wall. What would a real ball do differently?
First, a bet on plain descent.
Find the biggest step size where the ball still settles.
Bigger steps move faster along the floor. Watch the walls.
Give the ball momentum: reach the bottom within 30 steps.
The step size is locked at a small 0.02. β is how much speed the ball keeps from step to step.
Race three update rules down the same valley.
Plain descent, momentum and Adam, each with a good setting.
Momentum, Adam, optimizers.
Most real networks train with Adam or a close relative.
β = 0.9. The velocity is 2, the new slope is 1. What is the new velocity?
v = β × v + g = 0.9 × 2 + 1 = 2.8. Most of the old speed stays, and the new slope adds to it.
A small dip sits before the real bottom. Pick a β that gets past it.
Too little speed and the ball gets stuck. Too much and it jumps the fence.
Code: write the momentum update.
Fill the two gaps in the training loop, then Check to run it.
Three optimizers ran down this valley. Which path is momentum's?
A zigzags wall to wall: plain descent. B swings past the middle and curves back: momentum. C heads down with similar steps in both directions: Adam.
Run your code with your β.
On the dip and on the valley.

Lessons are interactive and need JavaScript. Turn it on to play them.