TwiddleLab / Artificial Intelligence / How Machines See and Hear

Fooling a Vision Model

Lesson 7 of 10 · 7 min · Free, no account needed

What you will learn

• An adversarial example is an input changed on purpose, often by a tiny amount, so a model gets it wrong.
• The attacker uses the gradient: it says which way to push each pixel to hurt the answer most. Random noise of the same size does almost nothing.
• Training on attacked examples makes a model more robust, but not unbreakable.
• Real attacks exist: stickers on road signs, printed glasses, hidden commands in sound.

Start the lesson

How the lesson goes

1. Researchers changed a panda photo so little that nobody could see it. The AI said gibbon, 99% sure. How?

   Try it on our digit reader first. Guess, then Check.

2. Turn up the attack until the reader stops seeing a 7.

   Each pixel moves at most by the strength, in the direction that hurts the answer most.

3. Is it just the amount of noise? Compare aimed and random noise of the same strength.

   Switch between the two.

4. This trick has a name.

   Tap to reveal.

   An input changed on purpose to fool a model is an adversarial example.

   The panda (Goodfellow and others, 2014) moved each pixel by under 1% of its range, aimed with the gradient.

   A model that keeps working under such attacks is robust.

5. Pixel nudge: fool the reader in 6 taps or fewer.

   Each tap pushes one pixel a quarter of the way along its arrow. Then Check.

   The arrows are the gradient: how much each pixel's change raises the reader's error. Pushing the biggest ones first breaks the answer with the fewest changes.

6. A defense: train the reader on attacked digits too. Does it hold up?

   Slide the attack strength and compare the two readers on 397 test digits.

7. Game: armor test. Fool the armored reader with as few taps as you can.

   Same 7, same nudges. Compare with your count against the normal reader.

8. Code: programmers attack their own code too. Find a score that breaks grade().

   Tap the number line, then Check.

   grade() trusts its input. A score outside 0 to 100 still gets a grade, which is nonsense. Checking inputs and testing odd cases makes code robust, just as attacked examples make a model robust.

9. These attacks were all shown in research. Which kind of model does each fool?

   Drag each card, then Check.

   Adversarial examples work on any model that follows a gradient from input to answer: vision models through pixels, speech models through sound samples.

10. Replay: attack a digit you draw.

    Draw a digit both readers get right, then press Attack.

Previous lesson: Reading Digits · Next lesson: Sound Is a Picture Too · Part of the course: How Machines See and Hear

Lessons are interactive and need JavaScript. Turn it on to play them.