Tokens: How a Model Reads

Lesson 4 of 12 · 7 min · Free, no account needed

What you will learn

A model reads tokens: pieces of text that can be a word, part of a word or a single letter.
A tokenizer is built by merges: fuse the most frequent pair of neighbours, again and again (byte-pair encoding).
Common words become one token. Rare words, typos and less common languages fall into many pieces and cost more.
Letters hide inside tokens, so counting letters is hard for a model and easy for code.

A model that writes essays got the r's in strawberry wrong. Why?
Make a guess: drag the marker to how many pieces you think the model sees.
Watch a tokenizer being built from letters.
Each merge fuses the most frequent pair of neighbours in the text. Drag the slider or press Play.
After 4 merges: which pair fuses next?
Tap the pair you think appears most often, then Check.
This is how every chatbot reads.
Tap to reveal.
The pieces are tokens. A token can be a whole word, part of a word, or one letter.
The program that cuts text into tokens is a tokenizer. It's built from merges: fuse the most frequent pair, repeat. This is called byte-pair encoding.
Real tokenizers keep about 50,000 to 200,000 tokens. The model only ever sees token numbers, never letters.
Tokenizer race: beat the BPE machine.
You have 6 merges. Fuse pairs until the critter chat needs as few tiles as the machine, or fewer.
One token or several?
Drag each word into a pile. Every drop shows how a bigger tokenizer cuts it. Then Check.
Break your tokenizer, then fix it a little.
Find a word your merges cut into 6 or more pieces. Then add one merge to it and Check.
Code: be Python and count the r's.
Python looks at every letter. Tap each r, then Check.
Use the merge list to cut "unbelievable".
Apply the merges in order in your head, then draw the cuts and Check.
Race: your tokenizer against a big one.
Type your name or tap words. Fewer tiles wins. Race at least three words.

Lessons are interactive and need JavaScript. Turn it on to play them.