Music and Voice Generation

Lesson 8 of 11 · 7 min · Free, no account needed

What you will learn

Sound is cut into short slices, audio tokens, often 50 per second.
A next-token model continues music or speech the same way a chatbot continues text, with a temperature setting.
Voice cloning needs only a few seconds of someone's voice, so a familiar voice on the phone is not proof.
Protect yourself with a second channel or a family code word.

A few seconds of your voice can be enough to clone it. How does a machine learn to talk like you?
First, a guess.
Continue the tune one note token at a time, like the model would.
The bars show the model's chances for the next note. Add 4 notes it finds likely.
Let the model write 12 notes. Find a temperature that is lively but not wild.
Low temperature always picks the favourite. High temperature lets rare notes through.
Sound becomes audio tokens, and the model predicts the next one.
The same trick that writes text writes music and speech.
Text model and sound model: what do they share?
Drag each chip into a column.
Both are next-token models with a temperature, trained on huge amounts of data. Only the tokens differ: pieces of words or short slices of sound.
Three calls in familiar voices. Stay safe on every one.
Any of these voices could be cloned. Tap what you would do.
Math: a model uses 50 sound tokens per second. How many tokens is a 3 minute song?
3 minutes = 180 seconds, and 180 × 50 = 9000 tokens. That is why music models need long context windows.
With temperature near 0, which note does the model pick after F, every time?
Tap a bar, then Check.
Temperature reshapes the chances as p^(1/T). As T shrinks toward 0 the biggest chance takes over completely, so the model always picks G, the 37% bar.
Your duet: the model continues your tune at your temperature.
Press Play and continue.

Lessons are interactive and need JavaScript. Turn it on to play them.