SIRAJ: -Hey, are you OK?
WOMAN: -I want to make music but I don't know how!
SIRAJ: -Hello World, it's Siraj!
In this episode, we're going to talk about the bleeding edge
in machine-generated music,
then write our own music generation script in TensorFlow
in just under 90 lines of Python.
I'm a big fan of Hans Zimmer.
-Don't leave Matthew McConaughey. No...
-But I don't have the time to put in 10,000 hours to be like Hans.
But just imagine a program where I can say
"I want a song that conveys the feelings of hope and wonder
while sailing the Seven Seas
but also the gaseous feeling of eating too many beans."
The program would convert that speech to text
and, for each word, it would find the associated vector.
It would then compare those vectors to vectors created
by a model trained on a labeled music data set
for a range of human emotions.
The output of the machine would then be a set of chords
that it thought best represented those emotions.
It would lower the barrier to entry for me to compose music.
Machine learning is a layer of intelligence
that is meant to extend our own.
-I'm still really bad at folding shirts.
A laundry-folding robot would be so dope!
DeepMind released a paper a few days ago called "WaveNet,"
which is now the state of the art in both
music generation and text-to-speech.
Traditionally, audio generation models are concatenative.
That means in order to generate speech from some text sample,
it utilizes a huge database of speech fragments
by picking a few and combining them
to create a full sentence. Same for music.
This works, but it's hard to incorporate things like
emotion and natural sound when the output is created
by putting a bunch of static fragments together.
Ideally, we could have audio generation
that is parametric, where all the info required
to generate audio is stored in the parameters
of the model that we give it.
That's what WaveNet is capable of.
Instead of generating an audio signal by passing
its output through signal processing algorithms,
it directly models the raw waveform of the audio signal.
Researchers don't really do that, but WaveNet did.
The model they used was a
convolutional neural network where each layer
had dilation factors that let its interconnectedness
grow exponentially the deeper the data
flowed through the model, and each generated sample
at each step was fed back into the network
to generate the next step.
Let's take a look at the computation graph for the model.
The input data, a single node, starts as a raw audio wave.
We first formate the wave so that it's better suited
for processing, then you encode it to produce a tensor
with a number of samples and a number of channels.
We feed that into the first layer of the convolutional network,
which reduces the number of channels for easier processing,
then through a stack of layers with dilated convolution.
We combine the outputs of all the layers
and increase the dimensionality to the
original number of channels.
Feed it all to a loss function,
which measures how well our training is going,
and, finally, the output is fed back into the network
to generate the wave of the next time step.
This process keeps repeating to generate
more and more audio.
This neural net was huge.
It took 90 minutes on their GPU cluster
to generate a single second of audio.
Instead, we're going to build a more simple
music generation script in TensorFlow.
We want to first import numpy as a scientific
computing library in Python,
then pandas as our data analytics library,
then, of course, tensorflow, our machine learning library.
tqdm will help us show a progress bar during training,
and finally midi_manipulation which is our helper library.
Let's start off with the first step:
our hyperparameters, or tuning knobs, for our model.
We're going to use a type of neural network
called a "restricted Boltzmann machine"
as our generative model.
An RBM is a two-layer neural net.
The first layer is visible and the next layer is hidden.
Nodes within layers are connected to nodes
in the next layer, but no two nodes of the same layer
are linked. That's the restriction part.
Each node decides whether to transmit any data it receives
to the next layer or not by making a random decision.
We want to first create a range of notes
that our model will generate, so we'll write variables
for our lowest note and our highest note,
as well as a note range which is the difference
between the two.
Then we'll write variables for the number of times,
the size of the visible layer, the number of hidden layers,
the number of training epochs, batch size,
and finally our learning rate which is a
TensorFlow constant since it always stays the same.
Next, we can write out variables.
We'll start off by creating a placeholder
which will serve as our gateway for
inputting data into our model.
Then a variable that will store the weight matrix,
or the connections between our two layers.
We also need two variables for biases:
one for the hidden layer and one for the visible layer.
We now want to create sample from our input named "x"
using our gibbs_sample() helper method.
Gibbs sampling is an algorithm that creates a sample
from a multivariate probability distribution.
It constructs a statistical model where each state
depends on the previous state, and uses randomness
to obtain a sample across the distribution it creates.
We'll get a sample of the hidden nodes as well,
starting from our original input,
and a sample of the hidden nodes starting
from our generated sample.
Then we'll update our weight value based on
the differences between the samples that we drew
and the original values.
We will run these three update steps for our weights,
hidden biases, and visible biases when we later run our Session,
and we can store all of these in our update variable.
Alright, it's time to start Session so we can run our graph.
We'll initialize our variables first.
We're going to reshape each song so they are vectorized
in a good training format, then we're going to
train our RBM on these samples one at a time.
Once the model is fully trained, we will make some music.
We run our Gibbs chain where the visible nodes are
initialized to 0 to generate sample,
and reshape the vector to be the proper format for playback.
Finally, we'll print out the generated chords.
♫♫
Dope! The state of the art in music generation
uses a CNN to generate raw waveforms parametrically
instead of concatenatively.
That algorithm requires a lot to compute right now,
so we can use an RBM to generate audio samples
based on training data easily,
and Gibbs sampling helps us create audio samples
during training by randomly obtaining them
from a generated probability distribution
based on the input data.
And so, the winner of the last coding challenge
is Rohan Verma! He created an image classifier
for astronomers to classify galaxies as either elliptical
or spiral. Badass of the week.
And the runner-up is Chih-Cheng Liang
for creating a bird classifier for biologists.
I bow to both of you.
The challenge for this video is to use the RBM script
to generate an audio sample with a happy, upbeat sound.
Post your code in the comments.
I'll judge them and announce them in my video
one week from now.
For now, I'm going to get more silver hair dye,
so thanks for watching.