My name is Steve Milton, as Lisa just said, and would like to welcome you all here. Would
absolutely love to thank CMJ as well as Microsoft for making this all possible.
Also we’d like to begin by introducing everybody else up on stage here. We have Charlie Whitney,
Dave Rife, and Kamil Nawratil. I almost got it.
Listen. This is who we are. Thought it would be great to give you guys a little bit of
background on the project. I am the founding partner of Listen. We are a creative and strategic
group just down the street.
And most of our work is focused, and many of the projects we do, are focused on the
intersection of music and marketing. The two projects that we’re going to talk about
today are part of an ongoing partnership that we have with Microsoft.
Delqa is the first project we’re going to look at, that we worked on with Matthew Dear.
And the second, something that’s actually launching this week, was a live show visual
system for Neon Indian.
When we think about music and technology and we think about how technology has really over
history shaped how musicians are able to express themselves and how audiences receive music
and then how we also define what is music.
Technology definitely plays a role in that. And this is true now more than ever, and is
core to the work that we’re doing with Microsoft.
Specifically our work involves developing and concepting ideas and projects like this,
so we can bring together musicians with technologists and talented folks like the people who are
here on stage today.
Delqa is a project here, we’re going to watch a video in just a minute which will
help set it up, and then Dave and Charlie are going to talk a bit about, uh, get under
the hood a little bit on this project.
But it was a project, it was an installation that went up at the New Inc. space, at the
New Museum, over the summer. And you know, I think I’m going to let the video speak
for itself.
Technology’s allowing for this openness. It’s this living, breathing sound sculpture.
You can play with it, and tweak it.
I wouldn’t make the song that I made for DELQA at any other point in my career or my
life because it only works for a space like that.
To have that opportunity, and to meet wonderful new people and learn from them and see what
they’re doing and get ideas that I wouldn’t have come up with, I mean, that’s the best
. . . that’s art.
We saw this project as an opportunity to create something that had never been done, pulling
together an amazing artist with a team of amazing artists using Microsoft’s technology.
Seeing the guys break it apart, I’m seeing the guts of the system. It’s a lot of math
and a lot of numbers. These guys are smart; they’re really smart.
The big idea that we were exploring was being able to step inside of Matthew’s music.
This piece is really about creating a sonic environment. So one of the ways that we’re
going to achieve that is by installing 40 loud speakers throughout the space.
Those will all be linked together into a spatial audio system that allows for a three-dimensional
field of sound.
You’ll be inside of all these different musical ideas that make up one beautiful composition.
We were able to leverage the Microsoft Kinect’s ability to track people’s movements spatially
through this experience.
The Kinect is everywhere in this thing. You’re almost always covered by a Kinect. We specifically
sourced this fabric material. It was opaque to the depth cameras, so we can actually read
the surface of the fabric, but it was transparent to the colored infrared cameras, so we can
actually see through it.
As people push on the walls and displace the membranes, they’re actually able to change
the quality of the music.
Different areas of the installation affect the music in different ways. Sometimes a user
will be changing the actual part that’s playing. Sometimes they’ll be introducing
new parts into the music.
We wanted every interaction to feel very powerful for the audience. So, if we take the drum
part for example, as they press in, the rhythmic density of it becomes much more complex, but
we’re also changing the tamboral qualities of those synthesized drum sounds.
All of these parameters allow the audience to have a multidimensional influence across
the composition.
So you could visit this piece on a number of occasions and you’ll hear something different
each time. It would never be the same if you went there twice.
Our desire was to create a new type of architecture, influenced by how we experience sound.
Doing interactive design is really about creating a magical experience, and having really powerful
things like a Kinect that help you create those experiences.
There’s so much that we can do with a project like this.
The more accessible these technologies become, the more people are able to use them and do
surprising things with them.
People want to make music, people want to be part of the music making process. Technology
allows that.
Your mind is totally free to wander. There’s no right or wrong answer. It’s just however
far you want to push it.
To be able to bend music, make music, and put that in different worlds, that’s what
makes this fun.
So, uh…that should help to give it a little bit of context around the project, what we
were up to and how that all came together. Now I think I’ll hand it over to Dave to
talk a little bit about the composition and the sound and light and even a bit more.
Check, check, check, we’re here. So I’m glad that video played. I can kind of be done,
now. But no, we’ll go a little bit under the hood – [To Kamil] yeah, you can go ahead.
I lead a company with Gabe Liberti, who is out there somewhere. Where are you Gabe? Raise
your hand so everyone knows—that’s Gabe Liberti!
We have a creative studio called Dave and Gabe. We do a lot of sound and light work
with immersive and interactive environments. I have a background in architectural acoustics,
and, how sound behaves in space.
Gabe has a really strong background in production, and you can see why we ended up doing a lot
of the work on this project.
If you go to the next slide, you’ll see some of our work. It all involves interaction,
it, most of it, involves sound and lights. Go ahead, go to the next one, oh good that
was out…
I think one of the first things that we talked about as a team, that entire team that you
saw in the video, was, what is this interaction going to feel like?
Like, how are you going to trigger these different moments in the sound, and how are you going
to create this composition based on what Matt already wrote.
And so, we really loved the idea of something being physical. Actually like physically touching
something, something like an interface, but something more architectural.
So we had these two different types of meshes. One of them was kind of “spandexey” like
this, and one was more like a net.
And as you pushed into it, you would change the composition, you would change different
parts of this music, and so based on how you’re pushing like this, and also where you are
in the space, like if you’re pushing on this side or that side, you would trigger
different sounds and different effects.
You saw in the video -- this is how it kind of works. So that’s great. We can go to
the next one. But very tactile, was what we were going with.
And then, at the same time, we were working with Matt on the composition. So Matt totally
got it right away. He met with the whole team and totally understood this would be rad as
an environment. I’ll create some music.
But how does it work in terms of making that music interactive? Because typically you’re
listening just over headphones or out of your speakers and you’re just taking it in as
the listener.
But what we were all going after was putting you inside of his music, so pulling his music
apart and surrounding you with it, but also giving you as a listener agency to change
what’s going on, to manipulate the sounds that are happening. [To Kamil] No, not yet
not yet.
So, what you see here is the Ableton session. So Matt created his composition in Ableton,
and then we all shared that session together.
And we worked with him to take specific tracks, so you’re seeing six different tracks here,
so six different parts of the music.
But we write multiple clips per track. Right? So there’s different energies within each
one of those tracks.
There’s different kind of sounds that are happening within that, but it’s all aligned
to one type of sound, so like, one might be a high hat, one might be a kick drum, one
might be arpeggios, that’s all kind of in the vertical there, and then within each one
of those are different sounds, so that when you push into the netting, you can trigger
those different sounds. [To Kamil] So if you go to the next one we should see that.
This is kind of the way that we made up the media nodes that were synthesizing some of
the drum sounds, so you see, as we push in, you get more rhythmically complex, a little
bit more dense patterns, and then as you push out all of that goes back away… next slide
And so I don’t know – go ahead and hit it again so it plays the movie – so you
can kind of see right in the middle, um, there’s a kick selector, and then this is the Microtonic
VSC. Right in the middle, you’ll see, right around here…somebody is starting to push
in to the music and you see that little arrow clicking down on the different tunes, and
you hear the different complexity that’s happening within the music.
I think it’s a little out of sync with the visuals, but you can also see within the Microtonic,
there’s different filters happening, and that’s all based on the real time information
about where people are pushing as well.
So this is a custom tool that we wrote within Max for Live that Yotam Mann, that you saw
in the video, wrote. In order to be able to make this music more interactive, and not
just like a static copy of what’s going on.
And then so now we have the interaction that we want to do, and we have the music and the
music is flexible. So now we wanted to reproduce it over a huge audio system that’s all around
the listener.
And so we had 44 loud speakers. And the point of this was not for loudness—it’s not
to make the thing really loud—it’s so that we can pull the tracks apart. We can
take the stems out, and give each its own place to be. And we can also give all of the
stems, or some of the stems, we give them trajectories, so there’s sounds that are
moving around the space all around you, and there’s some sounds that are moving based
on what you’re doing within the architecture.
So we laid it out in six different zones—A, B, C, D—so A, B, and C and D were kind of
four different instruments, and then E and F were when you climbed up into the middle
of the netting, you’d get this kind of drone type thing happening around your ears if you
were there, and if you weren’t there it wouldn’t be there. Go ahead.
And then this is kind of what it looked like in one zone. So we had ten loud speakers for
each of those quadrants, and some of the sounds would be really diffuse and be happening all
around you if you weren’t pushing into the net, and then if you pushed into the net it
would go local to where you were.
So you could really like spatialize where the sound was happening just based on what
you were doing, on top of all of the compositional stuff that was happening. And you already
saw that. I think that this is just a really funny clip because Gabe is in slow mo with
a speaker on his head. I had to show that, sorry. Keep going, oh there it is again. Wonderful.
And then in terms of the spatialization we used MaxMSP. We had some of the tracks within
the piece. These were not so much interactive, but they were always moving around you.
So what you’re looking at here on the right side of the screen is like a top down view
of all the loud speakers in the space, so you’re looking at the floor.
And then those green dots are actually sound sources, so that’s like particular parts
of his track that are always like slowly rotating in the space, and then we had these high hats,
which is kind of that crazy one, randomly going all around, and that ended up feeling
like it was like a fly kind of going all around your ear. It was super cool.
And this is kind of what it looked like to mix the piece. It’s really interesting to
mix a piece over a large spatial system like this, and also interactive, so we had to set
up in front of each one of the quadrants and get the local mix right and then sit and listen
to the overall piece and get the overall thing right, so it’s a ton of variables, but that’s
kind of what it looked like.
And then you’ll notice here, go ahead, that there’s lighting in the space too that was
also reactive to what you were doing in the space and it was also tied to the music, so
Ableton was kind of driving some of the lighting as well.
So what you’re seeing here is again a planned view overhead of the lighting system, a whole
bunch of DMX-addressable wash lighting that we hit the netting with. Go ahead
And then we used TouchDesigner to drive that. So we had these different color palettes within
touch for each part of the song. There were kind of three parts of the song in the timeline.
And then as you pushed in, you’ll see here, so this is the live installation and you’ll
see different color palettes happening, that’s based on people actually pushing and pulling.
This is the software working in real time. And then you’ll see, this is what it kind
of looked like to be there. And this is the arpeggio, as you can hear.
The lights go in and out, the lights also change color based on how far in you are,
and this particular instrument was an arpeggiator, so as you push in you get higher frequencies.
This is a binaural recording. If you all were listening over headphones right now, as Gabe
is pushing with his right hand and soon with his left you’d hear it spatialize to that
side and the other side.
And then this is what it looked like in the end. I don’t know how many of you saw it.
But again, around the perimeter you were pushing and pulling that kind of spandex mesh and
then in the middle you were encouraged to climb up in the top and kind of chill out
and listen to the entire piece.
I should also say that the point of this was so that everybody could manipulate these different
tracks around the room, but all contribute to the one composition together, so it wasn’t
like you were isolated if you were listening on one side, you would hear the entire piece
happening.
And I was just going to end on this to say that we used eight Kinects and Charlie here
who’s brilliant wrote a really rad application that he’s going to tell you about, but it
was all done over OSC over the network, so the Kinects and Cinder were driving Ableton,
Max for Live, and MaxMSP. And that was also driving TouchDesigner for the lights. Charlie,
take it away.
Alright. So I am Charlie Whitney and sometimes I go by Sharkbox. And I was responsible for
doing a lot of the programming that was taking the Kinect and taking this data, and then
shipping it out to the other pieces so we could make cool music.
I’m going to show you some protoypey kind of things because obviously we couldn’t
build this thing full scale. This was a really big installation process. It was fabricated
from scratch by our friends The Principals.
And so this is us in their shop. Their very messy shop at the time. And this is just a
piece of this fabric. So we tested a bunch of different fabrics, and we needed something
that felt really good.
So my background is in installation work. So I’m a coder but I do a lot of kind of
media art installation, and if you want people to touch something, it’s got to feel really
good.
So we had a couple other pieces of fabric that were maybe more readable by the Kinect,
but they didn’t feel as good. And if you’re in somewhere where it’s this completely
immersive thing with all of these 40 channels of sound, its non directional kind of, so
you don’t know where its coming from. So you really feel like you’re inside something.
So we wanted to encourage people to touch and explore this thing that you’re inside
of to make you really comfortable. So this is a single swatch of this fabric that we
had. We ended up stitching three of these together for the outer panels.
And these deform really easily. They’re really soft so that we can get these sort
of highlighted areas where you’re pushing out.
And like I said in the video, the holes in this are a weird size, where the Kinect 2
is a Time of Flight camera as opposed to the Kinect 1, which is more like laser grid. And
for whatever reason the size of the fabric is just… the holes are big enough so that
the depth camera hits em and stops.
But all the other cameras see right through them. And we had a really funny experience
when we were testing this where if you push too far, if you’re really stretching it
out you’re actually stretching the holes out and there becomes a point where if you
push it too far the holes actually became see through and all of a sudden the depth
camera would pop through and you would see a person behind there. It was really weird.
So we almost had to work around that but it’s a cool interaction – maybe something in
the future.
Here’s another pretty dirty prototype. This is one of the first things we did. So that
same piece of fabric we were just looking at. And there was even a video that Dave showed
earlier, just pushing on this one piece of fabric, and what you can see here is there’s
these 4 blue dots.
And this was our first attempt at getting differences in the fabric, like what happens
when you press up, down, left, and right. So what we’re literally doing is just sampling
the depth at the fabric underneath these blue points.
So on this, the top point is fully triggered. So we had a four-channel sound system hooked
up just to mess around with this and we were just seeing – can you do up, down, left,
right – what is actually possible? I don’t know if anybody had done as fine an articulation
of fabric, and it was a lot of testing to figure out what this thing could be.
We weren’t sure at the beginning. Cool.
So this is actually, this is Dave from above. This is something we were going to try to
do. We had this idea for these kind of cylindrical things that you would go inside of or be on
the outside of and we wanted to see if you could test the fabric deforming from up above.
These were going to be these really weird percussive-like almost drums we were going
to make, but it didn’t really work, so this disappeared, but it was a cool point of the
process. Actually could you go back to the blank one? I think that was a video? Alright,
no? Alright, keep going.
So this is where the Kinects were actually positioned. So this is just the front side
of it. There were four on the front and four on the back side, so this is just looking
from the front.
And you can see that we have two in the far corners, which were looking at that really
fine, that soft touchable mesh that I was talking about. And these two kind of in the
middle were just looking at the cargo net fabric in the middle.
And this is kind of what they were covering. So the blue ones are the ones that were covering
the fabric net, and then the pink is covering the cargo. So this is very approximate, I
just drew this, but it gives you an idea of how much coverage we really could get.
The Kinect 2 can see I think 120 degrees, maybe even a little bit more, so we were able
to get close and really wide with these, and get a really good amount of coverage.
And then we were able to get like an x, y position of where somebody was touching on
this fabric. We can get the x, we can kind of approximate the y, and we were trying to
get a z position, how far someone pushes into the fabric.
And it was a little bit tricky because we’re not hitting it head on. But what we found
was that if you just push a little bit it only deforms the fabric a tiny bit, but when
you start pushing huge, the whole fabric moves. So it was a much better metric to kind of
see how much fabric was moving instead of how far they were pushing it.
Because a bigger push would move more fabric. It’s kind of weird but I’ll show you an
example.
So this is a video that’s maybe playing… all right so this is actually an application
that you can download. We took some of our tools and made it into an app that you can
download. So you can see this pink box that’s happening – this is sort of the amount of
fabric that is being displaced. And then there’s a dot in the center, and this is sort of our
x, y position.
So up in the upper right hand corner we have the image from the depth camera. The left
side is the debug, what’s happening after all these sliders and filters are messed around
with.
And that thing down here is what is called our region of interest, so that’s what we’re
really looking at and trying to figure out like where is this depth. It’s just a little
snapshot. So this will actually output OSC, so if you’re a musician and you want to
do something with soft tracking you can just boot this up and it will start sending you
messages that you can map to Ableton or Max or whatever.
And it’s a little bit blue but this is the address, it’s http://github.com/cwhitney/DelqaTools
and it’s free. It’s open source. It’s pretty rad that Microsoft let us put that
up. So check it out.
Okay cool. Thanks guys. So…. yeah. There you go.
I think we’re going to open it up to questions but first we’re going to go take a look
at this Neon Indian project.
I think just worth saying about this one, we’re going to watch a little preview video
that’ll give you a sense of what is about to premiere with the Neon Indian show this
week, actually tomorrow at Webster Hall.
But really again, like with Matthew Dear, Alan from Neon Indian, we sat with him. We
had some conversations about how can the band take their show to the next level. Started
kicking around some ideas, and really arrived at what I think is a really great output.
Let’s play this video and then we can hop into it.
Short and sweet. But you get the picture. So, Kamil.
Hello? Thank you Steve. Hello everyone. My name is Kamil Nawratil. I’m the Creative
Director at VolvoxLabs. We’re based in Brooklyn. We specialize sort of in interactive installations
as well as content creation as well as kind of pushing all these ideas into the physical
world.
So what was interesting with this project is, so essentially, before we worked with
other DJs and musicians, we thought about how to recreate the virtual into the physical
world and add something on stage.
So, you know, digitally fabricate walls that we projection map or use screens that we project
through to add another dimension on stage.
But this one, we actually took it backwards and sort of used the Kinect sensors to recreate
the actual reality, the human geometry, and put it into our digital software.
So we called it “Reconstructing the Realities,” and essentially the project consists of three
elements, so first we’re going to use the Kinects to scan the environment where the
band is. So we’re going to look at the stage, what’s around, and use the Kinect to get
the geometry data to put in our software and use it to generate visuals.
This will then… obviously we’re also going to look at the band members to look at their
movements, get data from that, and project it back behind them as an extension of the
reality that they’re in.
Also we will be able to use artist content to sort of feed the color schemes and different
effects within the virtual system. So again, quickly, about the process. Scanning
– I will show in the bit, I have a little demo – but the Kinect has a really cool
feature where you can actually point it at something and retrieve geometrical data as
an object and sort of use it anywhere in other 3D packages to texture it, or spin around
it, shade it, so it’s really interesting.
The second part is looking at the musicians through the Kinect sensors. So we have five
Kinects on the stage. Each one generates particles and different instances within our software
which is TouchDesigner to drive the visuals and represent the music in real time.
Thirdly, we can get inputs from the musicians, so midi OSC data, doesn’t matter we can
get it in, and drive colors and shapes and scale of all these objects. That also ties
into the style of the band or the DJ that possibly can use this package later on.
And then at the end we combine it all together to recreate reality.
So as Steve mentioned, we sat down with Neon Indian, with Alan, and he gave us his references,
his visual aesthetic to inspire us a little bit and sort of add our style on top of it.
So what we came up with is three different styles within the show. He’s a big fan of
“glitchy” things, so there’s a lot of glitchiness. Maybe you saw it on Fallon last
night. There was a nice little glitchy river flowing.
But we’re also pushing the new technology so you can see these particle people walking
around, its going to be the band members, as well as virtual cameras spinning around
the stage, so it should be a pretty cool mix of different stuff.
So scans. I’ll show you a quick demo of what the Kinect can do.
Just to let you know, Kinect, the awesome thing about it is that Microsoft opened up
the SDK for it, so you can access pretty much any information that the Kinect provides,
so whether it’s the tracking of your face, tracking of your movement, sound tracking,
as well as scanning of the 3D environment, the environment where the Kinect sits.
What I’m using here is Fusion Explorer, so you can see the environment, there’s
the depth camera, the camera pose finder, but what it really does…
So this is how we’ll scan the stage, obviously it’s Dave, but we can look at room, and
stage and instruments, and we’ll essentially add that in front of all the dynamic simulation
particle people.
But the cool thing about this is you can actually walk around an object and get a 360 scan that
you can use later on. You can actually texture it, too, there’s a capture color mode.
We’re using the depth image to get information from space and generate different reactions
within our software to where the musicians are and where their hands are, and where we
are in space essentially. So Kinect is really great for that. It can actually tell you a
lot about the environment that you’re in.
And here you can see the effects based on the depth images and point cloud images. Point
cloud image is essentially the position of ever pixel that the camera looks at in x,
y, z. So here you can see all these instances attached to those points. It’s really interesting
how you can actually see the 3D geometry of the shapes that the camera is looking at.
And quickly about the set up on stage. So we have 5 Kinects. Alan, the main guy, we’re
looking at his whole silhouette. He’s very dynamic on stage, so we’re going to try
to capture his entire movement and reproject that as dynamic simulations.
All the other guys, we’ll focus on their movements, on the instruments, and whatever
they’re doing to create the sound, which is a little bit different than what we used
to do before with musicians and bands, mainly DJs, because since we’re looking at their
movements while they’re creating the music, this is already giving us sound-reactive visual,
which is different than when working with DJs where you’re trying to really syncopate
everything to the beat and the DJ just stands there really.
So we’re trying to make up for that with visuals. But this one, we’re actually getting
a freebie by looking through these sensors at the musicians and getting these sound-reactive
visuals.
So here you can see all the Kinects are looking now at a person in space. This is within TouchDesigner
software.
So we’re able to look at the points in the 3D environment and position them wherever
we are, so that during the performance we can actually move the virtual cameras from
one guy to another. Let’s say Drew at this point is making the most movements, we’d
probably cut to that to sort of show the situation and the action.
All that is happening on GPU. GPUs are getting super powerful these days and anything that’s
real time needs to be sent through that. So we developed a bunch of techniques to look
at the depth camera image and generate particles based on that. So we have hundreds of thousands
of objects and geometries happening at the same time from five people and running at
60 frames per seconds, so that’s really amazing.
So when Steve approached us he also mentioned that this will be touring and this should
be packaged so that anyone can tap into it, like I mentioned before where you can add
your content in and drive the visual style of your own show.
So we created this DJ interface where you can access everything from, change the virtual
camera positioning, change the colors, choose your tracks, and actually also play with the
Kinect cameras. You can see on top there, there’s five.
You can manipulate the depth image and what the camera can see in terms of distance. And
on the bottom you can see all the virtual cameras switchers, so whoever is interesting
at this point you can switch to that super quickly, as well you can set it to automatic
mode so it’s going to just switch around on its own.
So. Great job Kamil. Round of applause.