In the sixth century BC, Pythagoras-- yes,
that Pythagoras-- proposed a concept called the music
of the spheres to describe the proportional movements
of celestial bodies like the sun, moon, and planets.
This music is not thought of as being
literally audible but instead a mathematical concept.
Math and music are intrinsically connected.
The field of algorithmic composition
dates back to the early days of computer science.
Translation models take an existing non-musical medium,
like a picture, and translate it into a sound.
These are usually rule-based.
So a rule may be that if it sees a horizontal line in an image,
it will interpret that as a constant pitch,
whereas a vertical line represents an ascending scale.
Evolutionary models are based off of genetic algorithms.
Through mutation and natural selection,
different solutions evolve towards a suitable composition.
Then there's the learning model.
By providing it musical data, instead of rules,
we can let it learn for itself how to create music.
We're fast approaching the point where we'll no longer have
to wonder how Mozart or Jimi Hendrix would have composed
a piece on a certain topic.
We'll just be able to ask their algorithmic counterparts
ourselves.
Hello, world!
It's Siraj, and today we're going
to use deep learning to generate some jazz music.
The first attempt by anyone to use a computer to compose music
was by two American professors at the University of Illinois
Urbana-Champaign, Hiller and Issacson.
They programmed the university's Iliac computer
to compose music.
And it generated pitches using random numbers
before testing them according to the rules
of classical counterpoint.
So if a pitch didn't fit a piece,
another note was generated.
It also relied on probabilities via a Markov chain.
It used the past to determine the probabilities
of the future.
The first piece was completed in 1957
and was called the Iliac Suite for String Quartet.
Although it made headlines in Scientific American,
the musical establishment was pretty hostile to them.
They thought it undermined human creativity
and didn't include them in their journals
until after Hiller's death.
Nowadays, there are a ton of amazing generative programs
for composers to aid them when they compose music.
Like iTunes?
Let's understand this process by building a model to generate
jazz pieces using Keras.
We'll first examine the music we're
going to train our model on.
Our input data is going to be a collection of piano pieces
by American jazz musician, Pat Metheny, in MIDI format.
MIDI, or musical instrument digital interface,
is like the digital alphabet for music.
It contains a list of messages that
tell an electronic device, like a sound card,
how to generate a certain sound.
So it doesn't store actual sound itself, which
lends to a much smaller file.
Since these messages are a sequence,
we'll use a recurrent network as our sequence learning model.
For each MIDI file, we'll extract
the stream of nodes for both the melody and the harmony.
The harmony's chords accompany the melody's single notes.
Then we'll group them all together by measure number.
So each measure has its own grouping of chords
and this measure chord pair is what we'll
call our abstract grammars.
We'll vectorize these inputs by converting them
into binary matrices so we can feed them into our model.
Now we can build our model.
This is going to be a double stacked LSTM network.
So our computation graph will look like this.
The vectorize sequence of notes will be input
into the first LSTM cell.
Then we'll apply a dropout to help
ensure that the model generalized well.
And we'll do that process one more time.
Then we'll feed the data to our last fully connected layer,
labeled Dense.
Since every neuron in the previous layer
is connecting to every neuron in this layer,
it will mix all the learned signals together
so our prediction is truly based on the whole input sequence.
We'll lastly transform the result with a softmax
activation function into an output probability
for what is likely the next note in the sequence.
When we build our first LSTM layer, by default
it will only return the last vector, rather than
the entire sequence.
So we set return sequences to True
so that it returns the entire sequence, which
is necessary to be able to stack another LSTM later on.
Using two LSTM layers instead of one
allows for a more complex feature representation
of the input, which means more generalization ability.
And thus, a better prediction.
Recall that recurrent networks are essentially
like a series of feedforward networks
that are connected to each other.
The output of each and its hidden layer
is fed into the next one.
And when we back propagate, with each layer,
the magnitude of the gradient gets exponentially smaller,
which makes the steps also very small,
which results in a very slow learning
rate of the weights in the lower layers of a deep network.
This is the vanishing gradient problem
and LSTM recurrent nodes help solve that
by preserving the error that can be back propagated
through time and layers.
Let's look a little closer at this process,
but first I got to say,
[SINGING] L-S-T-M. Say it again.
L-S-T-M. Say it again.
When data goes in it forgets the BS.
Remembers the good stuff then outputs the rest.
L-S-T-M. Say it again.
An LSTM cell consists of three gates--
the input, forget, and output, as well as a cell state.
The cell state is like a conveyor belt.
It just lets memory flow across unchanged,
except for a few minor linear interactions.
These interactions are the gates.
We can add or remove memory from the cell
state regulated by them.
They optionally let memory through.
Each is a sigmoid neural net layer and a multiplication
operation.
The sigmoid outputs a value between 0 and 1,
which describes how much of each component
should be let through.
We'll represent each of the gates
with the following equations, where
w is the set of weights at each gate.
The way its internal memory changes
is similar to piping water through a pipe.
So think of memory as water.
It flows into a pipe.
If we want to change the memory flow,
this change is controlled by two valves.
The forget valve first.
If we shut it, no old memory will be kept.
If we keep it open, all old memory will pass through.
The other is the new memory valve.
New memory comes in through a t-shaped joint and merges
with the old memory.
And the amount of new memory that comes in
is controlled by this valve.
The input is in old memory and it
passes through the forget valve, which is actually
a multiplication operation.
The old memory hits the t-shaped joint pipe,
which represents a summation operation.
New and old memory merge through this operation.
In total, this updates the old memory to the new memory.
We'll define our loss function as categorical crossentropy.
The crossentropy between two probability distributions
measure the average number of bits
needed to identify an event from a set of possibilities.
Since our data is fed in sequences,
this measures the difference between the real next note
and our predicted next note.
We'll minimize this loss function
using rmsprop, which is an implementation
of stochastic gradient descent.
So we'll predict the next note in the sequence
over and over again until we have
a sequence of generated notes.
We'll translate this into MIDI format and write it to a file
so we can listen to it.
Let's hear what this sounds like.
[MUSIC PLAYING]
At least it's better than Kenny G. So we're all good.
So to break it down, we can generate music
using LSTM networks to predict sequences of notes.
LSTMs consist of three gates--
the input, forget, and output gate.
And we can think of these gates as valves controlling
how memory is stored in our network to eliminate vanishing
gradients.
The winner of the coding challenge from the last video
is the Vishal Batchu.
Not only did he perform multiple style transfer,
but he took it a step further by applying it to video, as well.
Wizard of the Week.
And the runner up is Michael Pelka.
He successfully performed multi-style transfer
through a clever matrix operation.
The coding challenge for this video
is to generate a music clip for a genre that you choose.
Remember, it's just a sequence of MIDI messages.
Post your github link in the comments,
and I'll announce the winner next video.
Please subscribe for more programming videos.
Check out this related video.
And for now, I've got to memorize memory cells.
So, thanks for watching.