TwiddleLab / Artificial Intelligence / How Language Models Work

Words as Points

Lesson 5 of 12 · 7 min · Free, no account needed

What you will learn

• A model turns every token into a list of numbers, its embedding. Two numbers make a point on a map.
• Words used in similar sentences end up close together: close means similar.
• Directions on the map can mean something: man to woman is about the same step as king to queen, on a toy map. Real models have hundreds of numbers and this works only roughly.

Start the lesson

How the lesson goes

1. To a model, cat and kitten are two different tokens. How could it know they're alike?

   Here's a map of words. Tap where you think "kitten" belongs.

2. Watch the map being learned.

   Every word starts with random numbers. Training pulls together words that appear in similar sentences. Drag the slider or press Play.

3. A new word: glorp. Where does it go?

   Read how glorp is used, then drag it onto the map. Its nearest words light up. Then Check.

4. The map has a name.

   Tap to reveal.

   The numbers that place a word are its embedding. Here two numbers make a point; real models use hundreds or thousands of numbers per token.

   Close points mean similar words: they show up in similar sentences. Nobody writes the meanings in by hand.

   This map is flattened to 2D so you can see it, and it's hand-made for the lesson.

5. Nearest neighbour hunt.

   A word glows: tap the word closest to it before the bar runs out. Six rounds.

6. Predict the map: drag each word into its cloud.

   The words are hidden. Drop each one where you think it sits, and it flies to its real spot. Then Check.

7. Draw an arrow from man to woman. Then move it to king.

   Draw the arrow, drag its tail onto king and see where the head points. Then Check.

8. Math: king - man + woman, with real numbers.

   Each word is a point, so each is an arrow from the corner. Drag the three arrows onto the grid, head to tail, starting with king. Then Check.

9. Here's your glorp. Which word is its nearest neighbour?

   Tap a word, then Check.

10. A mystery: where does "bank" go?

    Turn the dial from a river bank to a bank account. Watch the point for "bank".

Previous lesson: Tokens: How a Model Reads · Next lesson: Attention: Which Words Matter · Part of the course: How Language Models Work

Lessons are interactive and need JavaScript. Turn it on to play them.