TwiddleLab / Artificial Intelligence / Generative AI: Making New Things

Text to Image

Lesson 7 of 11 · 7 min · Free, no account needed

What you will learn

• Captions and pictures are trained to land near each other on one shared map, a text-image embedding.
• Guidance pulls every clean-up step toward the prompt's spot. Too weak ignores the words, too strong overdoes them.
• Closeness on the map is measured by the cosine of the angle between arrows.
• The other way round, a chat model cuts a photo into patches: each patch becomes an image token next to the words.

Start the lesson

How the lesson goes

1. You type 'a fluffy blue critter in the snow' and get exactly that. How do words steer static?

2. Put each caption next to its picture on the shared map.

   Drag a caption onto the slot under the picture it describes.

3. Turn up the guidance until the picture follows the words but still looks natural.

   Guidance strength sets how hard each clean-up step is pulled toward the prompt.

4. A shared map of words and pictures, plus guidance.

   That is how a prompt steers a diffusion model.

5. Build a prompt that makes the target critter.

   Tap one word in each row. The generator redraws as you go.

6. The other way: a chat model reads a photo as tokens. Make it exactly 16.

   The photo is cut into square patches. Each patch becomes one token.

7. Math: which caption is closest to the picture?

   Tap each caption to work out its cosine with the picture. Pick the biggest, then Check.

   b: (4 · 3 + 3 · 4) ÷ (5 · 5) = 24 ÷ 25 = 0.96, the biggest cosine, so the smallest angle. a gives 23 ÷ 25.5 = 0.90 and c gives 11 ÷ 25.5 = 0.43.

8. Which picture did 'a spiky green critter in the snow' make?

   Tap it, then Check.

   Each word of the prompt moves its spot on the map, so a good result matches all three: fur, colour and place.

9. Early image models often drew six fingers or garbled letters. Why?

   A diffusion model learns patterns of pixels, not facts like 'five fingers'. Hands and letters vary a lot from picture to picture, so early models got the look right but the count or spelling wrong. Newer models do much better with more data and bigger text encoders.

10. Studio rush: deliver three customer orders.

    Read each order, build the prompt, set the guidance, then Generate.

Previous lesson: From Pure Noise to Picture · Next lesson: Music and Voice Generation · Part of the course: Generative AI: Making New Things

Lessons are interactive and need JavaScript. Turn it on to play them.