Translator: Helen Chang Reviewer: David DeRuwe
Hi.
My name is Curtis Hawthorne.
I work with Google Brain on the Magenta project.
With the Magenta project, we're seeking to answer the questions
"Can machine learning be used to create compelling pieces of music and art?"
"If so, how?"
and "If not, why not?"
Now these are easy questions to ask
but surprisingly difficult to answer.
A truly compelling piece of music or art transcends a generated data point.
Music and art connects us to one another,
engages us, surprises us,
and provides the means of expression
for some of our most personal and powerful stories.
Can a machine learn to do that?
It turns out we aren't the first people to ask these questions.
People have been trying to compose music using algorithms for a long time.
Back in the 1700's,
musical dice games were very popular.
They allowed any player to compose a piece of music
just by rolling some dice.
Of course, the compositions that these games produced
were limited to the snippets of music written by the creator of the game,
so in the end, most compositions sounded fairly similar to one another.
One way to solve this problem of limited compositions
is to make the rules for producing the compositions much more complex.
As far back as the 1800's,
Ada Lovelace was writing down ideas
about how computers could one day be used
to create elaborate and scientific pieces of music.
In the 1950's, researchers at the University of Illinois
finally got a chance to try out some of those ideas,
and they used their ILLIAC super computer to write music for a string quartet.
The music that they produced was much more interesting and complex
than dice game music.
Researchers and musicians
have been exploring those ideas ever since.
However, this approach still has a fundamental limitation.
The computer can only write music
that follows the rules of the algorithm programmed into it.
If you've carefully written an algorithm for producing string quartet music,
that same algorithm isn't going to wake up one day
and decide to write some dubstep.
What if there was a way to have computers write music
the same way that humans write music?
So instead of using handcrafted algorithms,
the way humans write music is first we listen to music,
thousands and thousands of hours of music,
from the earliest stages of our lives,
subconsciously picking up on patterns and learning about different genres.
Then when we want to write something new,
we can draw on that experience and our inspiration
to write a new composition.
This idea of having computers learn
not from handcrafted algorithms
but from data
is called machine learning.
But how can a machine learn from data?
There are many ways that this could be done,
but let me give you one quick example.
Say you wanted to write a system that can tell you
whether a given photo was of a cat or of an elephant,
and let's say that you already had systems in place
that could recognize features such as fur, tusks, and eyes.
If you fed enough photos of elephants into this system,
only telling the system that these are photos of elephants,
over time, the system would learn that features like tusks and eyes
are highly correlated with an elephant being in the photo.
Similarly, if you fed enough photos of cats,
over time, the system would learn that fur and eyes
are highly correlated with a cat being in the photo.
Now, in this system, the important point
is that we didn't write an algorithm
that listed out the specific features for each animal.
Instead, the system learned on its own
which features correspond to which animal,
and it even learned that some features,
in this case eyes, correspond to both animals.
Now, in this example we just have a few of these kinds of nodes.
But if you take many of these kinds of nodes,
sometimes many thousands of these nodes,
and arrange them in multiple layers,
you create what's sometimes called an artificial neural network.
It's called this because it's loosely based on the way
that biological neurons work together in a human brain to solve problems.
Now, the system that we have here
has many flaws.
There are many situations where it won't work.
For example, not all cats have fur.
I'm sure you could come up with a long list of other examples.
That's why for real image recognition systems,
we take those neurons I was talking about,
we arrange them in multiple layers,
and then we let the system learn on its own
which features are important for recognizing objects and photos.
Usually the first few layers
learn to recognize things like lines and basic shapes.
Then intermediate layers are able to piece together
those lines and basic shapes,
and recognize more complex shapes and patterns,
and by the time that we get to the final few layers,
it's able to detect things like spirals,
something that looks kind of like a bird,
maybe some honeycomb in the lower left.
All right, so now we have a way for machines to learn based on data,
but how can we apply that machine learning to the creation of art?
Here's an example of art created by another team at Google.
They call their system "DeepDream."
It works by taking a neural network
that's trained to recognize objects and faces,
and it runs that network on an image in such a way
that it dreams up objects and faces that aren't actually there.
And in fact, there was a recent charity art show in San Francisco
where people paid thousands of dollars
to take home DeepDream-generated artwork.
So clearly, we're at a point where machine-generated art
is able to make some kind of human artistic connection.
All right, now, how do we apply machine learning and neural networks
to music creation?
One way to approach this problem is as a prediction problem.
So you can say, given a sequence of notes,
predict what the most likely next note is.
To do this, we'll need to use
a slightly different kind of neural network
than the one I've been talking about so far.
We'll need to use a recurrent neural network.
This type of network has access
to not only the information that it's currently being presented with,
the most recent note of the melody,
but it also has access to a representation of information in the past,
so the previous notes of the melody.
With this type of network, you can ask the question:
given a sequence of notes, what's the next note?
If you repeat that process
and ask it to keep predicting the next note,
you can generate a whole new melody.
Now we've got a way to train a machine to produce a melody,
and we just need some training data.
So we collected a library of songs.
Tens of thousands of songs, everything from Bach to Radiohead.
We took the melody lines from those songs,
and we fed them into the network, one note at a time.
And then, as a starting point,
we gave the network the first four notes of "Twinkle, Twinkle, Little Star"
(Singing first four notes)
and asked it to continue the melody.
And this is what it came up with.
(Computer generated music)
(Music ends)
All right.
(Laughter)
Go to the website.
You can listen to the rest of it there.
(Laughter)
All right, so that's pretty good,
but we wanted to explore ways
that we could make melody generation even better.
Focusing on just next step note prediction has a few important limitations.
One is long-term structure.
If you're only asking the model to predict the next note,
it has difficulty coming up with things we like to listen for in music,
like repeated motifs and longer melodic arcs.
Another problem is repetition.
If your training data has a lot of repeated notes,
and you can hear this, for example, even in Twinkle, Twinkle, Little Star,
(Singing first four notes)
then over time, the model might learn that given a particular note,
if you ask it to predict the next note,
the reasonable prediction might be just repeat that same note.
Similarly, if the last thing that happened was a rest or silence,
a reasonable prediction might be, just add another rest, extend the silence.
And this works well short-term,
but as you can imagine, it breaks down in the long-term.
So one of our interns this last summer, Natasha Jakes,
came up with the idea
of using a technique called "reinforcement learning"
to improve our models.
Reinforcement learning works a little different
than the training method I've been talking about so far.
Instead of showing the model a list and saying, "These examples are correct,"
instead we ask the model to produce some output, and we give it a reward,
mathematically speaking, based on how well it did.
This is conceptually similar to the idea
of a composition student and their teacher.
The student writes a composition,
and then the teacher gives them feedback on how good or bad the composition is.
So using this system,
over time, the model will learn to favor producing outputs
that receive a positive reward.
Of course, the important part of this
is we have to define what "sounds good" is.
We certainly didn't have time for a human to sit around and listen
to all the thousands upon thousands of melodies
that the model produces, and rate them.
So what we did is we found a composition text book,
and we just took the list of -
it listed some rules for what makes a good melody,
and we took that list and programmed it in.
Of course the first rule that it lists is "avoid excessive repeated notes,"
just one of the problems we were trying to solve.
OK, so let's see how well we did.
First, I'll play an example of what the model produced
before the reinforcement learning.
You'll notice the melody isn't very interesting,
and there's a long silence in the middle.
(Computer generated music)
(Music ends)
All right.
And here's what it sounds like after reinforcement learning.
(Computer generated music)
(Music ends)
All right. A lot more interesting.
After we posted that result online,
MIT Technology Review said
that that particular melody sounded a bit like a jingle
from the latest toothpaste campaign,
(Laughter)
which we take as a high compliment.
(Laughter)
This far, I've been talking about music,
but Magenta is interested in visual art as well.
A breakthrough in machine learning
has been the ability to transfer the style of one image to another
so, for example, you can see
what any given photo looks like painted in the style of Starry Night.
We wanted to take this idea a little further,
and so in collaboration with another team at Google Brain,
we developed a network that could learn multiple styles and mix them together.
This is a video of Peekaboo, my teammate's dog,
transformed to look like several different painting styles.
So you'll see the slider for Starry Night,
just got turned up to 100%,
and it's being mixed with a few other painting styles.
And then we'll remove the Starry Night style from the transformed image.
All right.
You can go to the next slide.
OK.
We're working on a bunch of other projects too,
but I don't have time to talk about all of them today.
So what we've got here is pretty good.
But remember what I talked about at the beginning,
about how music and art connects us to one another, engages us, surprises us?
Has what we've done here accomplished that?
Perhaps not on its own,
but we believe the key to Magenta's success
will be the community.
We recognize that music and art is too big of a problem
to tackle on our own.
So we're building a community of musicians, coders,
machine learning researchers to work on this together.
We've posted the code for our models on GitHub
and we encourage anyone to try it out, tweak it, use it, improve it
in ways that we wouldn't have thought of.
Here's a great example of how community can take a relatively simple beginning
and transform it into something really interesting.
Remember that tune from earlier?
(Singing first four notes)
When we posted that example online,
Jason Wynn, who studies Vietnamese traditional music,
wanted to play along with it.
So he and another artist collaborated and created this video.
(Musicians playing together with computer generated song)
(Music ends)
All right.
That's great.
We love how the tune that Magenta created
inspired these musicians to come together and create something new.
Next slide, please.
We wanted to expand on this idea of musicians interacting with Magenta,
so for the Neural Information Processing Systems Conference this past December,
we put together a demo
that allows the musician to play with Magenta in real time.
So in this video,
the electric piano sound you hear
will be the human performer playing a melody,
and then Magenta will respond in real time and continue that melody.
(Musician and computer taking turns playing)
(Music ends)
All right. Next slide, please.
We think these kinds of interactions
is where it gets really interesting with Magenta.
We think that machine learning applied to music and art will be similar
to when Rickenbacker and Gibson figured out the electric guitar.
They created this new musical instrument,
but all they wanted to do was come up with a way
to make the acoustic guitar sound a little bit louder.
They could have never imagined
what artists like Jimmy Hendrix or St. Vincent would end up doing with it.
Similarly, we think that the tools that machine learning researchers
will come up with for working with music and art
will be used by artists in ways
that those researchers could have never expected.
And I hope that some of those artists are right here today.
Everything that I've talked about is up on our website.
The code is all up on GitHub.
You can download all of these experiments and run them on your own computer.
See for yourself how they work.
Try them out.
Maybe you've got a great idea
for how Magenta can be used to create new music and art.
Go ahead, try it out!
I"m looking forward to seeing what you create.
Thank you.
(Applause)