>> Hi.
Good morning, everybody.
My name is Andrew Barrameda and I'm going to speak a little bit about Music21 and New
Applications to Score Following and Feature Extraction.
Well, first of all, Music 21 is a project developed by VMAD and it is [INDISTINCT] today,
that is the name of the project.
First of all let me do brief introduction.
You know, that during the last years, there has been evolution, you know, related to computer
and music and it has changed the way that we acquire music and share music.
Now you can download or buy a song just clicking a button and it also changed [INDISTINCT]
for example.
And composers changed their way to notate and their scores.
You can do it now using computers or for example, there are lots of [INDISTINCT] that analyze
sound like the first note that we--that we listened today and for example you can detect
which song is playing, a musician just using your iPhone or an application so that many
applications related to analyzing the sound and the music but what about scores?
There are no application or there is not an important change to the applications that
analyzes scores.
So, for this result, Music21 must [INDISTINCT] development.
Music21 is a software that--well, it's a toolkit for helping people to ask--to answer questions
and simply and query about music and analyzing the scores because for example--and what if--have
you ever asked in--how many--what is the most repeated note in a song?
Well, maybe not but you could take a song and then count the notes and say, "Okay.
This one is the note repeated."
Okay.
But what if I say what is the most repeated note by Truman or some composer?
Well, the answer is not that fast.
And--or for example this one [INDISTINCT] and so forth.
Well, this is--you can get the--these answers just running one or two lines of code with
Music21 because it analyzes all the scores that you would not get the solution very fast.
This program also has another part that is to compose.
And it allows you to compose songs that are [INDISTINCT] now and the composers can use
to improve that.
But if you want to do a song.
You want to write a song and taking a part from another song--you know, another one song.
Well, you can do it and just--in opening these songs with Music21 and taking the part that
you want and mixing them and then creating the final score if more than 10 lines of code.
This--let me show you one example of--what is the most repeated note for example by like
[INDISTINCT] of Japan.
Well, this is only an example of course and the most important thing is that now, having
this information, you can then analyze the [INDISTINCT] of these composers or whatever.
But what--how does it work?
How Music21 works?
It analyzes songs and [INDISTINCT] digital format and then let you use these data and
from using this data and then get all that you want.
Like for example, that I said before, whatever.
The most--the format that it Music21 supports are--well, the most common one.
There are more than this one but these are the most common like MusicXML, ABC, [INDISTINCT].
And then what you can do is to get songs with this format, then convert these one I think
to Music21 format, do whatever you want with the data and then if you want compare part
of the song, then put the song into the previous format.
And then if you want to show it, you can with many files as you [INDISTINCT] you can use
whatever.
But well, what can we do with Music21?
With all the applications that we can create using this program, for example if we can
use data, you could generate, for example, we could create an automatic generator of
other one, we can create a program at few rules, fewer money can [INDISTINCT] rules,
set a length to the score and the computer will generate a good score, you know, mistakes
and this could be, for example, used by students or whatever and…
>> For the composition owners.
>> So, for example, why not--we could create Braille [INDISTINCT] we have a song that we
want to convert into Braille.
We can get this song and analyze the data and then get the notes, the sharps, flat,
whatever, the rhythm and then just using some definitions, you can say this song--all these
character [INDISTINCT] This tool application are in the Music21 library if you're in directing.
And well, the application that I'm going to talk today is the score follower or page turning.
We wanted to create a software that shows you this screen.
A score and then slides the pages automatically and then you don’t have to do anything manually.
You can just play a song in the beginning until the end.
And so, this was [INDISTINCT] and how does it work well?
First the computer has to detect the sound, has to show you the scores and mix them and…
>> So, it works with audio, not just with media [INDISTINCT]
>> No, sorry.
No.
>> Okay.
>> With audio.
This is the challenge because [INDISTINCT] that's easy.
Okay.
And so, how it works?
Okay.
We need scores, microphone and a computer.
But we wanted to do as general as possible and we don’t--we do not want to use expensive
high-wire or microphones.
So, we decided to use only a laptop [INDISTINCT] that is one, so we have--we got [INDISTINCT]
it worked and we used the microphone of this laptop and it records the music, you know,
probably from a violin that we [INDISTINCT].
And then it shows you the screen and the text [INDISTINCT] with some--you're playing [INDISTINCT].
This is--and this is using only a modified sounds because--well, it has to detect [INDISTINCT]
properties of the signal and you can do that with [INDISTINCT] That is one of the problems
of this--of this software.
First, let me start with the score.
We need the score.
We need to show the score in the screen and--but we also have to analyze the score because
we need to know what are the notes inside the [INDISTINCT] length of the notes, whatever.
So, we [INDISTINCT] use as many songs--as many scores as possible, so we decided to
scan these scores and then we use the--an OMR system.
The OMRs and so forth that converts the image of our work into a data work but we use that
OMR system that recognizes the note and then translate--converts these notes into Music21
software notes.
So, we can use all the songs.
We just have to scan this.
Scan them--these songs and use the software and then we have the data of the notes and
then the rhythm of the notes and the [INDISTINCT] of the notes.
This [INDISTINCT].
The other one is the signal that, well, we're using a--the microphone of a laptop, so the
quality is very--it's very low.
And I used half of the whole data of the--of the audio, so we have to [INDISTINCT] to process
this signal.
In order to do that, we use, in this case, [INDISTINCT] function so that it works very
fast.
We have to [INDISTINCT] if we want to record within a few seconds the sound that we process--the
sound recording [INDISTINCT] and the sound record and processes the song again until
the end of the song.
So, we have to process this sound very fast because [INDISTINCT] a computer from, you
know, [INDISTINCT] with processing the sound because you can finish the song and then it
doesn’t make sense.
So, using this data [INDISTINCT] function to find the fundamental frequency of the signal
and with the fundamental frequency of the signal we can get the pitches of the sound
and also the rhythm and the length of the notes.
In this we have estimate which now is playing with the musician and the--well, in the end
of [INDISTINCT].
This is an example of [INDISTINCT] of the processing.
These are at the end.
We have processed the audio.
We have here the frequencies and then we have to decide when each note is [INDISTINCT] this
is a [INDISTINCT] or a [INDISTINCT] whatever.
And knowing the duration of each frequency, you can estimate that this is quarter note
for example or 16 note or whatever.
With this, we have the last step that is the margin.
Well, you have take into account different things.
First one is that we are using an OMR recognition, so it introduces some lines so the extraction
of the [INDISTINCT] is not perfect.
We are processing the audio using the microphone of a laptop so, the audio is not good.
And you don’t have to match the notes that we have from the score and the rhythm and
the notes that we have from the audio and the rhythm and then predict which one is the
part of this song that you're playing.
This would be an example and this software takes--divide this song in many different
parts and then it calculates which ones are the most similar to the score.
So, it will [INDISTINCT] and finally, depending on the last prediction, because if the last
prediction is clear, it doesn’t make sense if the next one is clear, you know.
So, [INDISTINCT] last prediction and probability of [INDISTINCT] half of the score [INDISTINCT]
which one is the one that [INDISTINCT] And then automatically, when you can see that
is--there is [INDISTINCT] like [INDISTINCT].
Let me show you an example of this.
This is only a--what?
This is a software.
I was playing the piano.
Sorry for the performance because I'm not a pianist but it's only a 20 seconds video.
And you'll see that I'm playing this song and when the software [INDISTINCT] that--essentially
it's detecting the position every time and things that [INDISTINCT] it's like the page.
>> So, when does it start the page?
>> When?
>> [INDISTINCT] when you're past the operation, right?
>> Yes.
Yes.
That is the [INDISTINCT] in this page and depending [INDISTINCT] the basis--the base
depending on the position of the [INDISTINCT] because this song is [INDISTINCT] of the musician
and if for example you just start, it turns the pages and slowly [INDISTINCT] slowly.
But if your location is here, the speed of the sliding is faster because you have to
read the next page as it does in the second page.
>> Oh, you can [INDISTINCT] to the audio again [INDISTINCT] It's going to--we're going to
get a little bit of feedback.
You know, can you play your speakers loud?
I think it's going to work right for this one because…
>> Yes.
>> [INDISTINCT] such a feedback with…
>> Do you have it plugged in?
>> Yeah.
>> [INDISTINCT] such feedback device.
>> [INDISTINCT] and well, as we said, [INDISTINCT] works with [INDISTINCT] songs so you're going
to play [INDISTINCT] at the same time because it doesn’t work.
And it would be, usually for example for orchestras if--and it doesn’t, if you put this software
[INDISTINCT] and if it work and then, [INDISTINCT] the melody is not--it's not good to detect
and well, you could connect--you can connect in someway [INDISTINCT] now we do this the
melody of the violins and when we can--the measure of the play and whatever.
And well, that's all.
Thank you very much.