TwiddleLab / Courses / Artificial Intelligence

Inside Deep Learning

Open up a deep network: backprop, optimizers and why training goes wrong.

Advanced · 10 lessons · Free, no account needed

Best after: Learning From Data, Vectors and Matrices

Start the course

Lessons

1. Many Neurons, Many Knobs (5 min): A layer is a group of neurons that all read the same inputs.
2. Activation Functions (7 min): Layers without a bend only ever make a straight line, however many you stack.
3. Backprop: Blame Flows Backwards (7 min): Backpropagation starts at the error and sends blame backwards, layer by layer.
4. The Chain Rule in a Network (7 min): A network is a chain of small steps. Each step has a local rate: how fast its output moves when its input moves.
5. Mini-Batches and Noise (7 min): A mini-batch is a small random group of examples used for one step. An epoch is one pass through all the data.
6. Smarter Steps: Momentum and Adam (7 min): In a narrow valley plain gradient descent zigzags: a step big enough for the floor is too big for the walls.
7. Fighting Overfitting (7 min): A big network can memorize its training data: zero training error, big error on new data.
8. Initialization and Exploding Gradients (7 min): Going back through a layer multiplies the blame by its local rate. Over many layers the rates multiply.
9. Reading Training Curves (7 min): A learning curve shows the loss by epoch, on training data and on validation data.
10. Project: Debug a Broken Network (8 min): Debug in order: read the curves, check the gradient bars, change one knob, train again and compare.

Lessons are interactive and need JavaScript. Turn it on to play them.