TwiddleLab / Artificial Intelligence / How Language Models Work

Counting Is a Model

Lesson 2 of 12 · 7 min · Free, no account needed

What you will learn

• The simplest language model counts which word follows which. That's a bigram model.
• Divide each count by the row total and you get probabilities: 3 out of 6 is 50%.
• Here training is just counting, and sampling is spinning a wheel cut by the bars.
• Change the training text and the model changes with it.

Start the lesson

How the lesson goes

1. Where could a machine get its bars from? It could just count.

   Here's a tiny song. Tap the word you think comes right after "love" most often.

2. Watch a machine read the song, two words at a time.

   Each pair of neighbours drops a pebble into the row of its first word. Drag the slider or press Play.

3. Now you train it: count four cells yourself.

   Tap every word that comes right after an underlined "i" or "the". Each tap drops a pebble. Then Check.

4. Let the counts write. Spin the wheel for every next word.

   The wheel is cut by the row of the last word: more pebbles, bigger slice. Spin until it finishes two lines.

5. You just built a real language model.

   Tap to reveal.

   A model that looks at one word to guess the next is a bigram model. "Bi" means two: it works with pairs.

   Each count divided by its row total is a probability: 3 out of 6 is 50%. The rows of counts become the bars.

   For this model, training is just counting. Writing is sampling: spin a wheel where taller bars get bigger slices.

6. Turn pebbles into shares.

   After "the": rain 3 pebbles, sun 1. Drag the bars to the right percentages, then Check.

7. Make the song sing your word.

   Build lines from the word chips and add them to the song until "me" is the tallest bar after "love". Fewer lines, more stars.

8. Code: the same table is a dictionary.

   Drag three tiles into the program in the right order, press Run, then Check. One tile isn't needed.

9. Count it yourself: how likely is "love" after "i"?

   Slide the marker to the share, then Check.

10. Before and after: the original song against your remix.

    Spin both wheels three times. What comes after "love" now?

Previous lesson: Guess the Next Word · Next lesson: More Context, Better Guesses · Part of the course: How Language Models Work

Lessons are interactive and need JavaScript. Turn it on to play them.