Activation Functions

Lesson 2 of 10 · 7 min · Free, no account needed

What you will learn

Layers without a bend only ever make a straight line, however many you stack.
An activation function bends each neuron's sum. ReLU keeps positives and turns negatives into 0.
Bends add up: a few ReLU neurons with the right weights can draw bumps, plateaus and stairs. Sigmoid squashes into 0 to 1.

This critter is happiest at 20 degrees. Can a network learn that bump?
Each blue dot is how happy it is at one temperature.
Two layers, no bend. Put the line through both rings.
Watch the formula as you turn the knobs.
Meet ReLU: it keeps positives and turns negatives into 0.
One neuron with a ReLU. Make it flat until 10 degrees, then climb to 10 at 20 degrees.
Three bent neurons. Give each a weight to draw the bump.
Their bends are at 10, 20 and 30 degrees. The output adds them up.
The bend has a name: an activation function.
Each neuron adds up its inputs, then the activation function bends the result.
Which neurons of your bump are switched off?
Three rounds, three temperatures. Tap every neuron that outputs 0, then Check.
Math: which of these always give a straight line?
A linear function inside a linear function is still linear. Sort them, and see each graph.
3(2x + 1) = 6x + 3, (x + 4) × 5 − 2 = 5x + 18, 2(3x) = 6x, x/2 + 7: all lines. ReLU and |x| have a corner, x × x curves.
This neuron uses ReLU. What does it output?
Sum: 3 × 2 + 2 × (−4), plus the bias 1.
The sum is 6 − 8 + 1 = −1. ReLU turns every negative into 0, so the neuron outputs 0.
Shape badge: draw three new shapes with the same three bends.
Tap a weight to change it (−2 to 2). Match the grey shape.

Lessons are interactive and need JavaScript. Turn it on to play them.