TwiddleLab / Artificial Intelligence / How Language Models Work
Attention: Which Words Matter
What you will learn
- Attention lets every word look at the other words and decide which ones help it make sense.
- Each arrow has an attention weight. The weights from one word add up to 100%.
- The same word can mean different things: attention mixes in its neighbours, so "bank" moves toward river or money.
- Attention is a soft lookup: instead of one exact key, it takes a weighted mix of everything.
How the lesson goes
The trophy didn't fit in the suitcase because it was too big. What was too big?
Tap the word that "it" means.
Draw where "it" looks.
Drag the bars: thicker arrow, more attention. They add up to 100%. Then Check to see the model's arrows.
Now make it "too small".
Tap the dashed word and swap it. Watch the thickest arrow.
This is attention.
Tap to reveal.
In every layer, each word asks the others: who here helps me make sense? That's attention.
How much a word looks at another is an attention weight. One word's weights add up to 100%.
The model mixes in the words it looks at, so the same token can end up meaning different things in different sentences.
Remember "bank", stuck between river and money?
Run attention on each sentence and watch the point for "bank" move.
Attention puzzles: five sentences.
Tap where the orange word looks most. Then its arrows appear. Can you get 5 of 5?
Make one word matter more.
Turn up how much "cold" matters to "it" until it gets about half of the attention. Then Check.
Code: a dictionary against attention.
Look up "cat", then "kitten". Press Run on both sides each time.
The cup fell off the shelf and it shattered.
Set the arrows from "it", then Check.
Speed round: where does it look?
Three sentences, a few seconds each. Drag the orange word onto the word it points to.
Previous lesson: Words as Points · Next lesson: How a Model Learns: Less Surprise · Part of the course: How Language Models Work
Lessons are interactive and need JavaScript. Turn it on to play them.
