Mini-Batches and Noise

Lesson 5 of 10 · 7 min · Free, no account needed

What you will learn

A mini-batch is a small random group of examples used for one step. An epoch is one pass through all the data.
Steps per epoch = examples ÷ batch size. Small batches give many cheap, noisy steps; the full data gives few exact ones.
SGD trains with small random batches. Shuffle the data first, or the batches are not fair samples.

1,000 critter photos. To take one step, should the network look at all of them?
Every option uses each photo exactly once. Only the size of a step's batch changes.
One careful step, or twenty quick ones?
Run both. Each uses the same 1,000 photos.
Find a batch size that gets the error below 0.1 in one pass.
Try a few sizes. The ball takes one step per batch.
Batches of 10 or 50 give 100 or 20 steps, each a fair guess of the true direction. Your size is saved for the race.
Batches, epochs and SGD.
These words are on every training log you will ever see.
This training ends far from the bottom. Find the fix.
Look at the photo belt. Then try the three fixes.
60,000 photos, batches of 100. How many steps in one epoch?
Steps per epoch = examples ÷ batch size = 60,000 ÷ 100 = 600.
Which batch size does each property describe?
Math: a poll is a mini-batch of voters.
How many people must each poll ask so that 19 of 20 polls land within 5 points of the truth?
Which curve was made with batches of 1 photo?
With one photo per step, each step trusts a single example, so the error jumps around. Curve C takes only 3 steps: one per epoch with the full batch.
Race the full-batch bot with your batch size.
Three epochs each. Lowest error wins.

Lessons are interactive and need JavaScript. Turn it on to play them.