TwiddleLab / Courses / Artificial Intelligence
How Language Models Work
See how a chatbot picks every next word, reads in tokens and pays attention.
Best after: How a Neural Network Thinks
Lessons
- Guess the Next Word (5 min): A language model looks at the words so far and gives every possible next word a probability.
- Counting Is a Model (7 min): The simplest language model counts which word follows which. That's a bigram model.
- More Context, Better Guesses (6 min): The words a model can look back at are its context window. A bigger window gives better guesses.
- Tokens: How a Model Reads (7 min): A model reads tokens: pieces of text that can be a word, part of a word or a single letter.
- Words as Points (7 min): A model turns every token into a list of numbers, its embedding. Two numbers make a point on a map.
- Attention: Which Words Matter (7 min): Attention lets every word look at the other words and decide which ones help it make sense.
- How a Model Learns: Less Surprise (7 min): A model learns by reading: guess the next word, see the real one, measure the surprise, nudge the weights. This is pretraining.
- Temperature: Boring or Wild (7 min): Temperature reshapes the bars before the pick: low makes the biggest slice bigger, high evens them out.
- Why It Makes Things Up (7 min): When a model makes up something that sounds real, it's called a hallucination.
- From Autocomplete to Assistant (7 min): A base model only continues documents. Asked a quiz question, it may write more quiz questions.
- What Happens When You Hit Send (9 min): The app wraps your message with a hidden instruction and the whole chat so far. The model has no other memory.
- Project: Name Generator for Critters (8 min): Your generator is a language model on letters: count pairs (or triples), then sample with a wheel.
Lessons are interactive and need JavaScript. Turn it on to play them.
