all right yeah so um my favorite type of
programming is attempting things that
I've never done before which is an extra
reason why I shouldn't be a full-time
software engineer cuz I love the whole
process of failing and learning from it
which i think is really important
especially admitting that in front of
hundreds of people so as that artist
residency I did a series where of
artworks that were called new romance
and I took a play it well it was a play
on Neuromancer by William Gibson
obviously but um the reason that I did
this project was because I wanted to
blend special-effects makeup and
Technology together to better visualize
what modern cybernetic augmentation
could look like and I did this in the
sense that this is digitalized makeup
and when I had the show to show people
this they were like whoa that's really
cool but it's also weird and I don't
want to go near it but they also didn't
know how I did it so I got to talk to
them about no BOTS because this is just
running on an Arduino and it's a simple
circuit that I soldered together and the
face is Angelina Fabro it's making you
oddly like familiar with it but you
can't put a finger on it um yeah so
weird art and Technology stuff like that
and so that leads me to what we're
talking about today um I do visuals
under the name ms paint ms paint this
was a show that I just played in January
out of festival called Magfest it has
30,000 attendees and like was
surprisingly not as scary as tech
conferences yeah when I started doing
visuals I was using tools that other
people had made already and I was like
why not make this harder on myself and
build my own tools so I do a lot of
these chiptune shows and I wanted
something to work for myself
the thing that I had been working with
before was called women and excuse me I
was using this for a while it's only
available for Mac what it is is it's an
old-school type of video synthesizer
that has like patch board stuff that you
could factor in and it was really cool
but it was very specific and I'm gonna
show you what it looks like
so it has a bunch of these patches so
that you can pick different kinds of
simulations of however they're altering
the waves and you can change it in here
and you can hook it up to a MIDI input
and save it and then you can get really
wild back here I don't even know I read
the I read the instructions a little bit
but I gave up because I was like that's
that's just way too much and so I was
like I gotta I gotta do something myself
right and then steer like why why would
you do that yourself um so I already was
doing visuals so it seems like valid
time and effort output to you know do it
myself and and I also love challenging
myself like I said I love trying things
that I've never done before so my
process going through when I make things
is probably similar to like how a lot of
other people put things through I think
of it separately I'm like okay cool so I
have this I had an idea of how I wanted
to control the input because I wanted to
look cooler than just like standing up
on stage with a laptop and pushing
buttons I was like I want to look even
cooler this thing this is called a MIDI
fighter
it is a MIDI input through USB there's a
bunch of buttons there's a potentiometer
inside there's buttons on the side and I
was like I would the way cooler up on
stage with one of these things than just
a laptop so I have my buttons and I know
I wanted to do something in the browser
so that I could build that and then I
was like alright so how am I going to do
this and then I'd figured out how to put
them together in a hard way essentially
and that was you know learning a whole
new way API so maybe we all know about
I'm gonna give you a little bit of a
history of the protocol and so it's an
acronym that stands for musical
instrument digital interface and it's a
way to connect devices that make it a
make and control sound so some people
think that like when I talk about MIDI
I'm gonna be talking about the sound
that gets like that but we're just
talking about the messages that
communicate with it and so it's a
protocol that allows electronic
instruments and synthesizers to
communicate meaning a user can control
multiple electronic instruments from one
synthesizer or a computer there used to
be no coherent way for manufacturers to
do this but then in 1981 at the Audio
Engineering Society convention in New
York Dave Smith proposed a rough version
of a universal digital interface that he
called a universal synthesizer and
interface or USI and he was like gonna
go out and meet all these people and not
that many people came so you know he was
basically like
here's my project and I want other
people to contribute to it and everybody
was like it's always been a problem
apparently except somebody that did show
up was equatorial kakehashi who was the
founder of roland at the time and so the
following year they displayed the fruits
of their labor and it was a
demonstration on how MIDI worked and how
it was born and then they went a
technical Grammy for it in 2013 which is
cool and yeah so how does MIDI work
specifically and the most important
thing to know about MIDI is that it is
based on the idea of message passing
between devices so like pieces of
equipment or software and so you have a
keyboard and a synthesizer and you would
like to record a sequence using the
sounds that are in the synthesizer and
so you connect them so that they can
communicate using that MIDI protocol and
start recording and when that happens
you play notes on the synthesizer and
all of the physical actions are
transmitted as MIDI messages to the
computer sequencing software which
records the messages and so many
messages are brief numeric descriptions
of an action which I'm going to show you
with the web MIDI API when I push the
buttons on them so that can be key
presses joystick Wiggles knob turns
they're all encoded and you know you are
interpreting what those messages mean so
when you hear the sound that you're
making it's not because of the MIDI it's
because of the messaging that the MIDI
is telling the synthesizer together
basically the computer isn't making the
or isn't recording the sound itself yeah
sorry
why is my slide deck yelling at me sorry
about that
oh so the concept of channels essential
to how much meeting messaging works and
that's an independent path over which
messages travel to their destination
there's sixteen channels per MIDI device
and a track on your sequencer program
plays one instrument over a single
channel when dealing with the musical
application maybe messages in their
track find their way to the instrument
that channel like I said we're not
dealing with music today we're dealing
with web MIDI and the MIDI a typical
typically enables synthesizers drum
machines you know like the music stuff
that we talked about but since it
doesn't transmit the audio if you wanted
to make a drum machine using the web
MIDI API you would just use the audio
tag simple so you could build your own
drum machine if you like the other kind
of thing that you can use it for is live
event controls for lighting and special
effects really it's like you can you can
do whatever you want
so the web MIDI API spec supports MIDI
protocol by allowing web apps to
enumerate and select input and output
devices on the client system and send
and receive MIDI messages and you're
able to utilize non music applications
as well as music ones by providing low
level access to the MIDI devices on the
user systems and the web MIDI API gives
you an output that tells you what kind
of device is connected and you can do so
much with it it's really really wild um
I've said this a lot I'm gonna skip over
it use the audio tag if you're wanting
to make music basically web MIDI is
really enabling a new class of
application that can respond to MIDI
input using external hardware so use it
along with other api's and from 43 and
up and you're you're ready to go okay
well so I'm going to show you some demos
now of how it works
all right now also ignore the
deprecation messages in the console
alright so where is my screen
alright so the way that you use MIDI is
you request MIDI access it's built into
Chrome so that's why I'm able to not
have anything else in here
yeah it requests access to the MIDI
device and this is how you can see if
the browser supports it if it doesn't
it'll tell you it does not
you
and then this is going to output the
the object that I was telling you about
before that talks to you about MIDI
inputs
and the MIDI outputs because not only is
a button press it's the press the hold
and the release depending on the MIDI
device that you have that outputs all of
that messaging and then this one also
the one bad thing about it is since it's
a potentiometer like every single
movement is just going to keep on giving
me messages to you so let me clear this
out and let's open one of these up
alright so in here you can see
that my MIDI device is a MIDI fighter
would be unless it's the next one
oh wait here we go
so this is a MIDI Fighter 3d it is made
by DJ tech tools this is not a sponsored
talk I just chose their device because I
like it and then when you go in a little
bit further there's a data
right here and it gives you an array
that identifies a lot of different
things it's based off of the spec that
you get for the MIDI controller but for
for me at least I know that the index 0
item in all of the data is contributing
to a specific button that is on the
device so that's how I'm able to control
the work that I'm going to show you in a
second and it also allows you to do fun
things like
hello so that's what I mean like you
don't have to do music you don't have to
do like games you can even just use it
to control simple elements on a webpage
all right
so what I built I decided to call Vinny
I originally was going to do like
shaders I was like that would've been
really cool but shaders are hard so
there's this really awesome library
called pixel sense made by Donald Hansen
and it's it essentially did exactly what
I was looking for so I asked him if I
could use it and adapt it to working
with a MIDI controller and he said yes
which is a nice thing to ask um yeah so
I'm gonna show you how that works
here is another example of what the MIDI
fighter is so it's full color
configurable arcade buttons total motion
control and three accesses it's it's
really cool people use it for drumming
like this but I used it for vide I do
want to give a warning in case anybody
instead of two flashing light if you
have that issue let me know and I will
give you a second to leave does anyone
have that too you don't okay cool so now
I'm going to give you a demo of
how this works
all right so this is it it is all in the
browser
and see that when you use the
potentiometer it affects on one axis the
rotation and on the other one it's scale
so it's really great for live music
because even if I'm not familiar with
the artists that I'm performing for I
can do it on the fly if I'm on beat okay
and then the other other functions that
are programmed into this is ugly color
so let me switch it to something cuter
I have a randomizer on the side that
allows me to keep on going through until
I can get a setup that I like and then
from there I'm also able to change the
width of the waves
so that it's very it's kind of hard to
see
but it allows for quick changes on the
fly which is really cool and I'll show
you how the code looks for that
it is mess
but um you know at the beginning we're
requesting that MIDI access like I was
saying I'm gonna make this bigger also
it's on github if you want to doctor it
for me
or you know perform yourself
ah command what P B thank you see it's
been so long I don't even write code at
my job anymore vp e
[Music]
all right so I'm not going to go over
all of the synthesizer code because it's
essentially just a lot of rgba waves
really MIDI is the star here and
everything in here is commented for
clarity and yeah so I have a lot of that
output values for the different buttons
I have an output for controlling the
potentiometer and the accelerometer or
the just the potentiometer because we
needed to round that to a more median
number otherwise it would have been all
over the place and very sensitive so we
are controlling it through these helper
functions which is just like averaging
out the frequency of
is all of the synthesizer stuff which is
pretty cool
and here is what we did with the changes
so this scale on the rotation is getting
that input from a potentiometer and then
I'm handling the messaging and honestly
like just randomizing everything cuz it
works it allows you to like check for
the specific buttons for changing the
waves it allows to track for the
potentiometer and then defining through
the scale and rotation so it's almost
all the way there the only issue is like
since it is a canvas animation if I
rotate it at a certain point in the
browser window it kind of rotates and
you can see the white background but
that is an easy fix
yeah tilt logic later I wrote this two
years ago there's not going to be a
later all right so the last thing I'm
gonna do is I'm gonna show you what it
looks like when I perform it live I
picked a song I'm gonna do like a minute
or two cool I'm not over I was so
nervous to talk to you today it has been
a while you would think that but stops
happening after but it does not all
right let's get out of here let me put a
song
yeah
[Music]
[Applause]
[Music]
yeah so the future and the future would
be ideally what I've done the past two
years if I did anything but since I
haven't and now I'm working on it again
I would love to have more custom MIDI
mapping in it so that people are able to
use something other than the MIDI
fighter so that you know people can plug
in whatever they want and play around
with it
excuse me I'd also like to add in
multiple visual options so instead of
just like canvas drawing web GL shaders
p5 3GS maybe just play in CSS animations
you want to do any of that you can I'm
going to try
and if you are interested in web MIDI
and want to try any of this yourself
there's a bunch of resources that I'll
leave up if you want to take a picture
it has the web MIDI API shim which I use
because it's it's easy the web MIDI API
hasn't really changed that much
there was the deprecation message in the
console because I hadn't updated that
code to follow the new spec but the new
spec has like a one-line fix that would
get rid of that so I probably should
have just fixed um yeah
Thanks
[Applause]