TwiddleLab / Artificial Intelligence / How Language Models Work

Project: Name Generator for Critters

Lesson 12 of 12 · 8 min · Free, no account needed

What you will learn

• Your generator is a language model on letters: count pairs (or triples), then sample with a wheel.
• Temperature trades boring names against odd ones. More memory makes names more real but copies the training set.
• Compare models by their surprise on names they never saw, and check the output for copies.

Start the lesson

How the lesson goes

1. A new species of critters needs 100 names. Build a machine for it.

   First, pick your training data.

2. Train it: feed it your names.

   Read the names in one by one, or all at once. Each letter pair drops a pebble.

3. Make 10 names, letter by letter.

   The wheel is cut by the pairs after the last letter.

4. Turn the temperature.

   Try it cold and hot. Where are names new but still easy to say?

5. What you built, in AI words.

   Tap to reveal.

   Your names are the training set. Counting letter pairs is a bigram model, on letters instead of words.

   Spinning the wheel is sampling, and temperature reshapes the wheel.

   Looking further back is a bigger context window: more real, but closer to copying.

6. Upgrade the machine: give it more memory.

   Drag the right block into the upgrade slot and compare the names. Then Check.

7. Pairs or triples: which model is less surprised?

   Both read real names they never trained on. Tap the better model, then Check.

8. Name forge: 10 names, at least 8 new and easy to say, no copies.

   Set the memory and the temperature, then Make 10 names. Win in 3 runs for all stars.

9. Pick your top 5. Catch the copies.

   Drag five names into your top 5 and the copies into their own pile. Then Check.

10. Code: the counting part of your generator in Python.

    Put four lines in order. One tile doesn't belong. Run it on your names, then Check.

11. A model with lots of memory wrote 20 names. Which are copies?

    Tap every copy of a training name, then Check.

12. Draw a critter for your best name. Then share your card.

    Draw on the white square. Your top 5 and settings are already on the card.

Previous lesson: What Happens When You Hit Send · Part of the course: How Language Models Work

Lessons are interactive and need JavaScript. Turn it on to play them.