How a Model Learns: Less Surprise

Lesson 7 of 12 · 7 min · Free, no account needed

What you will learn

A model learns by reading: guess the next word, see the real one, measure the surprise, nudge the weights. This is pretraining.
Surprise is the loss: -log2(p) bits for the probability p it gave the real word. Halve the chance, add one bit.
The weights, or parameters, are billions of numbers. Facts like "Paris is the capital of France" end up stored in them.

Nobody typed facts into ChatGPT. So where does it keep them?
Tap the box where you think a model keeps this fact.
You are the model. Guess, then feel the surprise.
For each gap, drag the bars, then reveal the real word. The lower your bar for it, the bigger the surprise.
Now a model reads a billion pages.
After every word it nudges its weights toward less surprise. Drag the slider or press Play.
Turn up the reading and watch a fact appear.
Turn the dial for pages read, all the way to a billion.
So that's where facts live.
Tap to reveal.
Reading huge amounts of text and learning to guess the next word is pretraining.
The surprise at the real word, -log2(p) in bits, is the model's loss. Training rolls it downhill, like the valley in course 1.
The numbers it nudges are its parameters (weights). Your bet's answer: the fact lives in the weights, not in a database.
Build one training step for words.
Drag four blocks into the loop in the right order. One block doesn't belong. Run, then Check.
Be less surprised than the rookie.
Read a short story as the model. Set your bars before each word. The rookie spreads its bars evenly. Lower total surprise wins.
Math: how many bits is a 1-in-8 surprise?
1/2 is 1 bit, 1/4 is 2 bits. Slide the marker for 1/8, then Check.
Which word would surprise a trained model the most?
Tap one word, then Check.
Your story score against the class and a trained model.
Show the scores.

Lessons are interactive and need JavaScript. Turn it on to play them.