[MUSIC]
My name is Kelly Snook,
I'm a music producer, I'm a musician,
and I have a PhD in aerospace engineering from Stanford University.
I was a NASA research scientist for 19 years,
and now I do music technology.
I am, or at the University of Brighton where I design and build new musical instruments,
and I have a fab lab where I teach people how to build almost anything,
how to make almost anything.
I'm really interested in this interaction between
the physical world and the digital world and especially music.
I'm very, very interested in science and music and using music to investigate reality.
I work in a new field that I invented myself called investigative music,
like investigative reporting, but where you use music to investigate reality,
so to represent information with sound and can communicate information with sound.
[MUSIC] One of the things
I've been doing over the last few years is
working on developing the Mi.Mu gloves which I'm wearing right here.
This is a way of mapping gesture to sound or visuals,
or really anything that a computer can control.
The artist or dancer can decide for themselves how they want their movements and
their gestures to control what you see on stage or what you hear in the studio,
or really anything in between.
They were created together with Imogen Heap and we were
looking for a way together for her to be able to express herself onstage,
which is quite a complex thing.
If you've ever seen her music or seen her perform on stage,
she's really doing a lot.
She spent a lot of time in the studio crafting
her sounds and really getting exactly the sound she wants,
but on stage it's not always as easy to
communicate that in a way that the audience can identify with.
She just wanted something a little more physical and little a more
visceral than just being behind her computer.
Even physical, the way the gloves are
designed was around how Imogen wanted to be able to perform.
She wanted her fingertips to be free so she could play the piano,
she wanted the palms to be free so she could clap because she does a lot of looping.
She needed the gloves to be not only like strong and robust,
where she could use them as a musical instrument and use them on stage,
but they also had to be beautiful because she's wearing them as part of her outfit.
This, we're all very interesting constraints,
and in development of a new technology.
We [LAUGHTER] were in a unique position of
having someone so visible and so virtuosic in her
music-making using this brand new technology before it was even done being developed.
This very rapid development of technology alongside true artistry,
and that coupling, the demands of design,
design, design, invent, perform.
I was 19 years at NASA and is a much slower process to
create something new and launch it in the NASA world or what it is in the music world,
especially with someone like Imogen Heap who
was really pushing the boundaries of technology.
But what we found is as we were developing these gloves,
that she's not the only one that wants to be able to to have superpowers in her hands.
So as a team, we started looking at trying to make them available more
broadly to the public and make them available
to everyone that might want to use them in whatever capacity.
That's a different problem designing a new interface
for a lot of people as
opposed to just one person that wants to do something very specific.
It's been a really interesting journey to have so many of the members of our team,
artists themselves, both performing and recording artists and producers.
We really understand what a user might want because we are ourselves users and
so I also have the opportunity to write and record and perform with the gloves.
I think that makes a difference in the technology in the end and this coupling and
the creative side of technology is really exciting to be part of.
Then to have them go straight from there onstage with
Ariana Grande I was under the stage when she was using them onstage,
and experimental technology isn't necessary designed for the demands of a super pop tour.
They're a MIDI controller essentially,
but it's not just MIDI or software.
We have the hardware,
but we've also been developing the software alongside the hardware.
The software actually is where the power is in doing
this mapping and the machine learning
for learning an individual's range of
motion and individual's way of making different postures with the hand.
We have that all in our software,
so when you go to create your music first,
before you create your music,
you create your instrument to the make the music with.
So there's a whole new meta-level of creativity and in fact,
almost a whole new art form that's developing now around instrument creation.
Even when you have the basics of the instrument,
like with a pair of gloves you still have to create the instrument itself,
which is the mappings from the gestures to the sounds and how you want
that to feel and sound like they are a MIDI controller,
but our software also sends out OSC which is much more powerful,
I think than MIDI in certain ways.
You can really create
custom control of almost any piece of music or visual software these days.
If you can output OSC and MIDI,
that's basically all you need to control anything.
Literally any Arduino patch or any Max for Live patch,
any Max patch any PD patch,
any piece of software that you can do MIDI mapping to,
any piece of hardware that you can control with MIDI.
For example, Ariana Grande,
we used TC-Helicon Voice Live Touch 2, one of the harmonizers.
You just have Ableton Live sending out MIDI,
sending it straight to this piece of hardware
that then harmonizes and sends the audio back in.
So really, you can route, however, you like,
and eventually, probably our software,
will have some actual audio engine in it,
but right now it's literally just the bridge between the gesture and MIDI or OSC.
[MUSIC]
People ask me a lot what's the future of music?
A lot of time you get invited to speak about this topic.
Everybody wants to know the future of music and I have an answer to
that which is essentially just my inner hopes and dreams.
Which is at least in part a return to the past when music
was considered a tool for investigating reality.
Actually, in medieval times,
music was part of the quadrivium.
It was mathematics, geometry, astronomy,
and music as scientific tools.
Well, they didn't have science yet,
but as tools for investigating reality,
and I love this.
In those days, actually music wasn't even an art,
it wasn't considered an art,
it wasn't a tool for self-expression or
emotional expression or any real expression other than
to mimic or investigate the music of the spheres and the music of
the human condition and the three kinds of music were music of the spheres,
Musica Humana, and instrumental music were
instrumental music was the only audible kind of music of the three,
and it was meant only to mimic the other two.
I would like to see a future and I am creating a future where it's possible to use music,
return of music that's that spot in
the quadrivium where it's used to express mathematics,
it's used to express the beauty in nature that we see with our eyes,
but often we don't get to hear because it's inaccessible to us, for example,
I'm really interested in investigating the music of the spheres,
which we can't hear because a,
it's not audible, but secondly,
it's representing something that's at it's such a different scale from our daily life,
we can't access it.
To be able to use music the way Kepler did.
Kepler used music to discover the laws of motion of the planets and the solar system,
and that's how astronomy was launched,
through music, and people don't really know that.
Not only do they not know that,
but they don't have access to this music and we haven't heard it yet.
Four hundred years ago,
Kepler challenged us to use our arts to do that.
He said, even if it takes a 100 years.
Four hundred years later,
and we're just now technologically at the point where we can hear this music.
I want music of the future to be
the music of the spheres and the music of the human condition.
I want for people to think of music as a way of expressing actual reality, science.
I want to return to there not being a difference between science and music.
The instrument that I'm building,
which is Latin for harmony.
It's called Concordia.
It's a scientific instrument and a musical instrument.
I think when you play Concordia,
you won't be able to tell whether you're
playing a game or you're playing a musical instrument,
or you're investigating solar the system.
It'll all be part of the same experience and I
think that's a really exciting aspect of the future of music.
In fact, there's a whole economy built into this instrument as well.
Even the economies around how we participate in music,
whether it's as a creator or as an observer,
I hope and I'm really devoting my life to the future of
music being non-commoditized, non-industrialized.
There are economies around it,
but they're economies of participation in a common good,
rather than trying to treat music as if
it's a cup of coffee, or it's something that you buy and sell,
and the burden of everyone's salary and livelihood is on somebody's output.
This to me, I think really needs complete re-investigation in terms
of the way we think about music and how we use it in our society and how we pay for it.
The instrument that I'm building and the future of music,
I hope we'll look a lot different economically as well.
Hopefully will be sustainable.
I think it can be but completely different from how it is now.
That's my vision and my hope.
We do it all the time in our heads, I mean,
a lot of us anyway,
who see and hear at the same time.
We're using the capacity to analyze
our environment with every sense that we have at our disposal. Whatever we have.
Some people have fewer senses than others,
but whatever we use our brains synthesizes together.
The idea of taking information and sending it over sound is
not that much different than taking information and sending it into a plot or a graph,
it's that just that we don't have that paradigm as well-developed because
science and ideas have been expressed for so many hundreds of years in written form.
It's only very recently in terms of our history of
communicating ideas with each other that we are
able to do so in such a multimedia contexts.
We haven't developed a paradigm for how to
encode information in sound and then decode it.
We haven't trained ourselves how to do that either.
We're really far behind the curve in terms
of developing ways of doing this kind of encoding,
but there are certain things we've been doing for a long time that do that.
Like a stethoscope, the way a doctor listens to a heartbeat,
there's a lot of information that a doctor gets just from listening to that heartbeat,
a lot of different things.
Then the doctor is really using the capacity for analyzing timbre, not just rhythm,
but timbre and lots of other elements of sound that are very highly honed,
but maybe not necessarily ever articulated,
it's just through experience.
If we could find ways,
especially I think of using some of these aspects of music that our brains
are particularly good at decomposing with sound, especially music.
The musical sound comes into your brain.
It immediately gets routed to all different places and your brain
processes melody in one place and
rhythm in another place and timbre in yet another place,
and all of this is happening in parallel,
and it's all like tied in with emotions and memories.
There's all of this capacity that we have for literally deconstructing.
Sound comes in, what is sound?
It's just one wave.
Basically a one-dimensional wave.
We can take that and in our head,
we can hear that,
say it's like a stereo image,
at least if you've been mixing for a while,
you can decompose that into 100 different sounds and tracks and things.
If you train yourself to listen,
you can tell where the sound is placed,
you can tell which instruments they are,
you can tell how many of them there are,
you can tell whether it's course,
you can tell if it's reverb,
you can tell what kind of space there in.
All of this information is coming in on this one
vibrating thing and then our brains can take it apart.
What if we used each one of those channels as the carrier for actual information?
Well, that wouldn't be too hard to then train ourselves to
understand scientifically in addition to feeling it the way we feel music.
It's just a matter of being clever about how you do the encoding.
This is the part where there's a whole field right
now called data sonification where people are trying to figure out how to do that.
What are the equivalence of say,
if you see a plot of temperature,
we've been doing this for so long that if you see something blue,
you know that's cold and red is hot.
That's arbitrary, but that's the paradigm.
What is that equivalent in sound?
We don't have one yet, but you could imagine that there could be,
and timbre is almost like an analog for color and rhythm is an analog for,
I don't know something else visually.
If we could find those ways of trying
these analogs between the sound world and the visual world,
and then actually extrapolating and taking it further to optimize
the two senses or even more senses smell and haptic feedback,
trying to really make use of those properties.
Like our hearing, we can hear orders of magnitude all at the same time,
much more than the orders of magnitude quantitatively,
visually that we can process.
If we can put information into that we can
represent scale in much different ways with sound than we can with visuals.
You can really mess with time as well,
and you can learn a lot from a very short sound.
If I didn't have these gloves on,
and if I were to knock on this surface and this surface and your eyes were closed,
in that teeny-tiny space of time,
you would be able to imagine what that surface was made out of just because you know.
But that's a timbre analysis that happens in a very short amount of time and
you would never be able to represent that visually really in such a short amount of time.
We don't know how to do it,
I guess, or to the same level as visualization,
but sonification is very rapidly developing because
our technologies are rapidly developing so hopefully,
this instrument that I'm building, the Concordia instrument,
will help us to explore these ways in which we can optimize
sending information over sound and sending information
visually and how we synthesize that in our brains.
Lots of people are working on this,
but this will be a fun way,
an instrument that uses data that's already so beautiful and musical,
and to really play with it until we get it right.
[MUSIC] Such a great experience with Berklee Online.
I started looking into it at a time when I had taught myself
music production from the Pro Tools manual and from Digi Designs on the phone.
These are the days before online help.
They actually got to know me pretty well, "Oh it's
Kelly Snook calling again.
She needs to know which end of the microphone to plug into the desk."
[LAUGHTER] When I first started making music,
the only thing that got me through was,
"Kelly, you're a rocket scientist you can figure this out."
But a lot of the things that I figured out,
I figured out the boneheaded way to do it and
just from what I could make sense of the documentation.
I didn't have any experience in studios and I didn't have
any context for what I was trying to do,
just what I wanted to hear.
I made it up as I was going along.
By the time I found Berklee Online, I thought, "Oh,
I can actually learn the right way to do this and keep my job at NASA."
So I was trying to do this part time and I was really skeptical,
in those days, there wasn't really a model like there is now for distance education.
It was especially hard to imagine how would you do music production
and listening and getting experience? Would it work?
And it was really expensive for me at the time.
I was really skeptical and I think one of
the first classes I took was critical listening.
I was just astonished with how effective it was,
how well designed the exercises were and the interactivity.
I mean, now it's normal but to be able
to interact with classmates and interact with the teachers,
and to be able to reproduce things that were happening somewhere else in your own studio,
I was well impressed.
I'm glad that I also
waited until I had tried it for so long myself.
Because it really meant a lot more to me.
Oh, that's how you do that. Oh, that's how you do that.
Oh, that's what that means.
If I hadn't struggled by myself at first it may have not been so revolutionary. But it just
stepped up my capacities in the studio.
Yeah, I really loved it.
[MUSIC]