More Context, Better Guesses

Lesson 3 of 12 · 6 min · Free, no account needed

What you will learn

The words a model can look back at are its context window. A bigger window gives better guesses.
A counting table needs a row for every combination: vocabulary to the power of the window. Almost all rows stay empty.
With a big window, a counting model can only copy its training text. Real models learn patterns instead.

Two robots counted the same song. One writes better. Which?
Press Write on each robot a few times, then tap the one whose lines make more sense.
The difference: robot A looks back 1 word, robot B looks back 2.
Turn the window from 1 to 5 words. Watch what it writes and what happens to its table.
How big does the table get?
Slide the marker to your guess, then Check.
Here's what you found.
Tap to reveal.
How many words a model looks back at is its context window.
A counting model that looks back several words is an n-gram model. Its table needs vocabulary to the power of the window rows, and almost every row stays empty.
With too big a window it memorizes and copies: the overfitting you met in course 1. A good model generalizes: it writes new text that still makes sense.
Sweet spot: make your robot write new lines that hold together.
Pick a window and press Write. Win with 2 or more green lines in one go.
Who wrote it? Drag each line to its robot.
Window 1, your robot and window 4 each wrote some of these. Then Check.
Math: 50 words, a window of 3. Build the number of rows.
Drag tiles into the base and the power, then Check. Watch the tower.
A robot with a window of 6 read one song. Tap every line it copied.
Compare its lines with the song, tap the copies, then Check.
A mystery: pull the word strawberry apart.
Real models don't use tables of words. So what do they read?

Lessons are interactive and need JavaScript. Turn it on to play them.