TwiddleLab / Artificial Intelligence / How Machines See and Hear
Sound Is a Picture Too
What you will learn
- A microphone measures air pressure thousands of times a second. Each measurement is a sample; how many per second is the sample rate.
- Cut the sound into short slices, split each slice into its pitches, stack the columns: a spectrogram.
- A spectrogram is an image, so the same filters and networks that see can also hear.
- 1 second at 16,000 samples per second is 16,000 numbers.
How the lesson goes
You say 'Hey' and your phone wakes up. Sound is just air wiggling. Can it be a picture?
These three pictures are three sounds. Tap the one you think is a whistle, then Check.
First, sound into numbers. How often must a microphone measure?
Raise the samples per wiggle until the dots follow the wave.
Now numbers into a picture. Chop a voice into slices.
Press Run. Play works too.
The words for it.
Tap to reveal.
Each measurement of the wave is a sample. Samples per second is the sample rate: 16,000 is common for speech.
Slices turned into columns of pitches make a spectrogram: time across, pitch up, loudness as brightness.
It's an image, so the filters from this course can find patterns in it, like a word.
Draw a sound: a whistle that rises.
Paint on the empty spectrogram and press Play to hear it.
Match each picture to its sound.
Play them if you like, then drag each picture to its name and Check.
A whistle is close to one pure pitch, so it's one line. A drum hit is short and noisy, so it covers many pitches for a moment: a column. A voice has a base pitch plus many multiples of it: parallel stripes.
A noisy room, 3 seconds. Find the moment someone says 'hey'.
Tap the picture, then Check.
In the spectrogram the voice shows as several stripes stacked above each other. The drum is a set of columns, the whistle a single line.
Math: a phone records 3 seconds at 16,000 samples per second. How many numbers is that?
Sample rate × seconds = samples. 16,000 × 3 = 48,000 numbers, one per measurement.
Game: your whistle is now a wake word. Tune the detector.
It must wake on all 3 of your sounds and never on the others.
Previous lesson: Fooling a Vision Model · Next lesson: Faces, Cameras and Fairness · Part of the course: How Machines See and Hear
Lessons are interactive and need JavaScript. Turn it on to play them.
