yeah okay so my talk is titled the
browser as a modular networked video
synthesizer and everything I'm going to
talk about is online in the form of sort
of a code editor and graphic surrender
at this at this website Hydra editor dot
glitch dot me and this is sort of a tool
I've been building but really it's more
of an investigation of of three
different things that might seem kind of
separate collaboration in the creation
of live visuals possibilities of
peer-to-peer technology on the web and
feedback loops and has also become a
platform for live coding visuals in
JavaScript so I'm going to start with a
reference from the 1970s this is the
satellite arts projects by kick Galilei
and che robe into it and what they did
was using using satellites in 1977 they
streamed video from two different places
and composited them on top of each other
and what they said was that this marks
the first time that the geographically
dispersed electronic image was
contextualized as a live immersive place
where artists and others could convene
and co-create together on a scale that
could be as cultural inclusive as
desired and so this is before the
internet existed this is just satellites
and they applied to get access to this
the satellite and to send video from one
place to another and it's sort of this
utopian idea that as soon as maybe if we
just put two images on top of each other
we create this third place it's not here
it's not there but it's this other place
that's some hybrid of these these two
places and I want to stay here for a
second because I think we still use this
idea of a place in talking a lot of
about the web it's a website as if
you're going somewhere or it's an
address a webpage just this idea that
when you access a website you're you're
going to a place and it's sort of this
analogy that doesn't necessarily have
that much to do with the underlying
technologies like if I go to a web site
my browser sending a message to a server
somewhere that server sending messages
back to me I'm sending messages back but
it's not there's not that there's
actually a place that I am necessarily
and then also this idea that it's live
and immersive I think Andres is talk it
was a good example of maybe we're just
getting right now to this idea of a live
immersive place on the web so I want to
jump a little to now which is um and
talk about WebRTC which is a browser
protocol for streaming video audio and
data directly between clients so this
means that now in the browser we can be
a little less dependent on a centralized
server this server here is this pink
thing or some some way that all of these
clients can talk to each other but then
WebRTC says that I can as a client with
a browser directly send video audio and
data to another person who's connected
and in this diagram I'm I'm sure a lot
of people have seen it before but but
says maybe with this kind of these kinds
of decentralized protocols we can go
away from the left-hand image which is
more essentialized Network and go
towards the right-hand image which is
more of a decentralized network where
where let's of different people can
connect and exchange information in
different ways and there's a lot of
exciting projects I think being built on
these this type of protocol caught for
example web tour in speaker browser the
dot protocol and stuff like that and I
just kind of got interested in this
protocol and started doing experiment
sending video balancing video around the
internet and as I was doing that I sort
of started to feel like this
this thing on the Ray of this network
looked a little something like this
thing which is another reference from
the 70s because for some reason a lot of
my references are from the 70s maybe
it's cuz I'm from San Francisco
originally but anyways so what this is
is an analog video synthesizer and and
basically it's a bunch of boxes some
boxes create a signal by the time that
signal gets to the screen on the right
the pattern of the signal creates a
pattern and the boxes are designed in
such a way that many of the boxes can
receive a signal and transform it and
then so the order in which you connect
the boxes will affect what comes out on
the the final screen and this is similar
in analog modular synthesis for audio as
well as analog modular synthesis for
video and so I thought what if we kind
of treat this thing this thing kind of
like like this thing instead of thinking
of of a website as actually a place what
if we think about it as a bouncing
signal that we can play around with and
so so I started experimenting with this
idea about two and a half years ago or
something and so what this is is
basically here I'm using WebRTC to send
a video signal between various browser
windows so here I'm sending this in this
video to this window and compositing
them with each other and then I'm
sending this video in this video here
and compositing them with each other
over here and these are all happened to
be on the same computer to kind of show
the idea but they could be you know when
one window could be in China and one
somewhere else one here but this is I
don't know why this was interesting to
me it was just
let me explore this idea and because I
didn't know what I wanted to do with it
instead of actually making a user
interface as I was experimenting with it
I just live coded in the console kind of
playing around with different things and
seeing what happened just with this sort
of routing video streams and throughout
the Internet
[Music]
yeah
and so Daniel sand in the creator of
that synthesizer that I was just showing
key he said the modules are designed to
maximize the possibility of
interconnection thereby maximizing the
number of possible modifications of the
image and so the idea his idea is that
the more of those boxes that you have in
the modular synthesizer the more you can
route signals maybe the more
possibilities you have so now I'm going
to move into the the tool that I've been
building to experiment with this this
idea
it's called Hydra and it's at that link
I showed earlier called Hydra editor
glitch that me and so it's designed to
let people experiment with with what I
was just showing you not just within by
sending video to different windows but
also just all within the same video with
it within the same window I mean and so
I'm actually gonna get rid of what's on
the screen and so the basic unit within
this is is an oscillator
okay so here we have an oscillator which
is like one of those boxes in the
modular synth world and then in order to
see it I need to connect it to a screen
so here's I connected the oscillator to
an output and you can think of in this
case this is a JavaScript library that
I've created for working with this model
of sort of video synthesis and and you
can think of each function as if it's
one of those boxes that we're seeing in
the modular synthesis world so now that
I have an oscillator I can give it
certain parameters like frequency sync
and also separate the colors somewhat
and then I can also start to transform
the signal so psy can make it rotate so
here I can make it rotate more
I can make it into a kaleidoscope
for example or I can pixelate it
[Music]
and so so here sort of the analogies are
each each function is is one of the
boxes in the modular synthesizer each
parameter is kind of like a knob and
then the chaining of the functions acts
like the patching cables for people more
from who are familiar with my dealer
synthesizers and this is this is
interesting to me this is all running on
WebGL and this chain of functions
renders to a single shader and I was
interested I think a lot of abstractions
for dealing with graphics either talk
about canvas like which you use a lot of
drawing metaphors so there's a canvas
you draw a shape you draw another shape
or they use sort of 3d metaphors like a
lot of what we just saw which is a box
lighting a scene and so here I'm using
this other metaphor which is the video
synthesis and transforming a signal
metaphor and so with this if you want to
repeat something instead of writing a
loop and saying I'm gonna do it now a
hundred times I just apply a
transformation so here I could do repeat
and so I I could do it two times or I
could do it 200 times and it's not more
problem it's not more intensive for the
process for the processor at all because
they're just transformations to the
coordinates directly on the graphics
card and the result outputs a pixel
color and so that's all well and good
but this is just sort of a single chain
of things and so what I was really
interested in for that mess of cables
and how things were out into each other
and so what I can do here is I can have
various various sort of output screens
and so here and I can also use different
sources and
I'm gonna use my webcam here so they're
there you actually want to put this over
there and that over here and so now I
have I have my two my two screens and I
can then also blend them together so I
have one output called a zero and one
called oh one so then they can now say
I'm gonna use a zero I'm gonna blend it
with a one and then I'm gonna output it
- OH - and so then there I am and
there's some different sort of blending
operations here wait I'll make this a
little and so one other thing that I
sort of took from modular synthesizers
either what happened what's happening
now is I'm combining two colors and the
two colors of the two images on the Left
create the colors on the right but I
could also use the colors of one thing
to affect the coordinates of the other
thing and so that in the analog world is
called modulation and so I can I do that
so now you it's the colors of the thing
on the top are affecting the coordinates
of my camera so it ends up with this
kind of were P thing and so hi everyone
I repeat myself and so this is just
something I I made this to experiment
with and a lot of and also back to the
image showing all the windows still if
anyone else is connected to this website
I can use what they're doing as an input
to what I'm doing which is sort of just
an experiment of of what happens when we
combine images in this way and then some
of the other things I've experimented
with of this this is like using the
screen as an input to itself when
creating feedback and this sort of
trippy thing oi here I'm using my
Twitter feed and and also sources and
the software can either be someone else
who's connected they can be a camera it
can be any browser window or it can be
any application window or it can be a
canvas element and so here here this is
my Twitter feed with some video effects
applied to it or this I use a live
stream of Time Square and a live stream
of an aquarium and then live coded the
mix between the two and and this for me
it's really just an ongoing
investigation of trying something out
and seeing what happens and trying
something else out
and another experiment this was using a
chromium with no background and so I
could actually just use my desktop as
input to this video synthesizer but with
without without having windows so that
was just a fun experiment I think
and so one of the interesting things to
me with this sort of setup is to be able
to make it easy to experiment with
feedback which in in the video world
feedback is where you have a camera and
the output of the camera goes to a
screen and then the camera is looking at
that screen itself and so there's no at
some point there's no starting point
because the image of the camera is on
the screen and the image of the screen
is what's coming out of the camera which
is similar to recursion in programming
or some of Escher's asurs drawings or in
the 10th century the snake that eats its
own tail so it's the this idea of things
that that don't necessarily have a start
so I'm going to show a very quick
example of that here it's also can be
useful for making fractals and so here
I'm just gonna output a shape can you
see my code or should I make it bigger
it's good okay so here here I have a
shape and then I'm gonna blend the
output of that with with itself so the
output here is called a zero we'll just
trust me on that I guess
so I'm gonna blend the output with
itself and we don't actually see any
difference because we're drawing a
triangle and then we're mixing it with
itself again which is a triangle but I
could after each time I mix the whole
thing repeat repeat it and then mix it
with itself again and so to do that with
this I could just do repeat and we get a
sort of fractal like thing which is
something that at different scales is a
repetition of itself and
it's something that's I'm just playing
around with it's really interesting with
this kind of set of obstructions is to
be able to in only four functions kind
of describe describe a fractal we're
gonna do one more little little demo
[Music]
I'm going to start with the camera again
hi that's my face very big and then I'm
gonna blend it with the output and then
I'm gonna use that modulate function
that I just showed you where the colors
of one thing affect the coordinates of
another thing and so here
so I get this sort of liquidy thing and
they could add a little more contrast
maybe let's see maybe a little more and
so it turns out that that feedback is
useful for modeling MA for modeling
things like fluid dynamics as well
because you can think of the colors of
the texture as sort of a representation
of pressure and then on each each loop
sort of effect the system using using
colors so yeah some things I've made
like experimenting with this just this
pretty simple simple setup
also it can be useful for creating
reaction diffusion like like patterns
so one thing that the creator of that
modular synthesizer that I showed at the
beginning talked about he talked about
the idea of this feedback loop which has
had sometimes like a mystical mystical
implications at different moments in
history but he says the reason you put
yourself in a feedback loop is so you
can learn something or rather learn to
do something you know you do something
in the results get processed somehow
perhaps just by yourself just looking at
it and you do another thing and the
results of that get process and your
actions can cause constantly be tuned or
corrected or refined by what you see and
and something that happens with playing
with these systems that can be kind of
delicate is that even though I know
exactly what each function does at the
moment of playing with it a lot of times
a really unexpected thing can happen
like like in this example if I make one
of these variables just change one
number maybe it makes it really it goes
away now and so it can be really
delicate and so being able to live code
it or - I'm actually executing parts of
the code at a time and being able to
code it see the results code it see the
results to me is is very similar to
putting yourself in that feedback loop
with which in an analog world is like
patching cables together and so part of
this is that since I put this online
about a year ago a lot of people have
started to use it and I still feel like
I'm learning to use it myself and so one
of the things I wanted to do is make it
easy for people to share to share what
they're doing and so there's a Twitter
bot uh-huh and basically this is it kind
of instead of having to pay for hosting
and have people upload images they're
basically directly from the editor we
can upload an image to the Twitter bot
so here and I'm gonna say J s comm
Hawaii yeah and then now it should
upload what I do
did to the Twitter bot let's see there
we are and so what happens is when you
when you click on a link here then it
opens the editor with the actual code
that someone used to make it and so so
to me since I started making this about
a year ago it's become now there's
probably like 200 people using it and so
part of it is that the software also
becomes a community and part of it is
how the things that I've been building
into the software can help people share
what they're doing and so so in the
Twitter bot you can also see this what
this happened came from what someone
else did earlier and this came what so
from what someone else did earlier and
this is also something I perform I'm
live with it there's a community of
people who do something called live
coding which people do within the tech
world but also they do it more within a
music of the music world that which is
write code in front of people to make to
make audio visuals and so there's a
website of people who are interested in
this kind of thing of of live coding as
a performer as an audio-visual
performance and so and I view some of it
as sort of a direct descendant of the
the video synth idea of putting yourself
in a feedback loop this is one of the
creators or the original the first
people to do live coding as a
performance thing and he had a live
coding band called slub in the in 2003
and and said he was heavily influenced
by reading paul Cleese pedagogical
sketch I'm not going to read this
booklet but
but the idea of live coding is sort of
an exploration in which you're trying
something out the computer does
something you respond the computer does
something else and you're sharing that
with a with a live audience so thank you
[Applause]