[Music]
hi everyone thank you for having me
my name is Olivia Jack my talk is
entitled the browser as a modular
networked video synthesizer hopefully by
the end of the talk it'll be clear or
more clear what I mean by that
everything I'm gonna be talking about is
online at this website Hydra editor
glitch taught me and it's sort of an
ongoing set of research and
experimentation and creation that I've
been I've been working on for the last
year and a half and so Hydra is it's an
investigation of collaboration in the
creation of live visuals possibilities
of peer-to-peer technology on the web
feedback loops and it's also a platform
for live coding visuals in in JavaScript
and so one of the inspirations for this
this project is a project from 1977
called the satellite arts project by
Kate Galloway and Cher Shari Rabinowitz
and in the 70s they got access to a
satellite and were able to stream video
live between two places at once and this
is in the 70s you know before the
internet before Skype and and they
started experimenting with this just
sort of streaming video and putting
images on top of each other
mixing them together overlaying them and
and seeing what what happened when they
started experimenting with this and one
of the things they talked about with
their project is this marks the first
time that the geographically dispersed
electronic image was conduct
contextualized as a live immersive place
we're artists and others could convene
and co-create together
on a scale that could be as culturally
inclusive as desired and I want to focus
on the live immersive place piece of
this because I think all they're doing
they're sending data between two places
right and they're talking about how just
by putting these two images on top of
each other and seeing them live that
it's creating this space that's not
there and it's not there it's sort of
some third third space that's a
combination of these two spaces and I
think now with the web we talk about we
use this kind of metaphor of a place a
lot like a website it's a site it has an
address you go to the address you're at
the website and it might just seem kind
of like nitpicky but it's also it's it's
kind of talking about a way that we're
engaging with these technologies and and
but what's really happening when I go to
a website you know my my computer is
sending a message to a server somewhere
the server is responding with with a
message and so I'm not anywhere when I
go to a web site but but we have these
sort of ideas about how we talk about
the web and how we talk about how we
engage with the web that to me it's just
sort of interesting to be aware of them
and start to dissect them and especially
just in the 70s this was like wow what
do you mean we can see live video from
two different places at once and what
what are the possibilities of this and I
think sometimes when when technology
just comes out is when people are often
the most experimental with it and so
fast forward to now there's this
protocol called web RTC real-time web
streaming that lets you to browser
protocol for streaming video audio and
data directly between clients and so
what's in one of the interesting
WebRTC is that although you need a
signaling server to sort of set up
initial handshakes between clients once
clients are connected they're basically
just sending video or audio or data
between one another and about four or
five years ago I started experimenting
with this this protocol making sort of
little video chat applications and it
sort of was just kind of amazed that
it's not that hard in JavaScript to like
send video everywhere and so so I'm
still kind of interested in this
protocol and I think lately there's been
a lot of exciting developments with this
such as web torrent like BitTorrent
running on the web or Beeker browser and
different sort of peer-to-peer
peer-to-peer applications of this is
saying what happens if we don't
necessarily depend on a server for
everything or on honest on
centralization for everything and so so
this this diagram kind of gets thrown
around a lot but on the left is like a
really centralized network and towards
the middle a more decentralized network
and on the right a more distributed
network and one thing that WebRTC
enables is possibly more things like on
the right but one thing that's been sort
of interesting to me is seeing a lot of
peer-to-peer technology that often
although it's use it it's it has the
opportunity to connect people like the
diagram on the right often what happens
is sort of replicating a centralized
database such as with Bitcoin or
something you're you're creating a
centralized database of transactions and
so everyone still sort of receives the
same information and in some ways it's
recreating the experience of a
centralized database and so
experimenting with this video streaming
I got interested in sort of different
ways of of thinking of using this kind
of client-side peer-to-peer network and
so one of the things that really
inspired me sending video around the
Internet is analog video synthesizers
and so what's an analog video
synthesizer this is an example also from
the 70s it's the Sandin image processor
created in 1971 by Daniel Sandin and
basically each of these boxes either
generates a series of electrical signals
or transforms a series of electrical
signals and when the when the signal
gets to the screen it creates a pattern
and then the order in which you connect
the boxes sort of creates different
possibilities for modifying and changing
the image and so what I was interested
in is is kind of I sort of seeing
similarities between this thing where we
can send lots of video streams around
and this thing over here which is
sending lots of signals around and the
way and the order in which which we're
sending them can create different
creative possibilities and this is
really I had no like specific goal in
mind but this is sort of like oh I just
want to play with these web protocols
and and it's not in this case not really
thinking of a website as a place or a
site but thinking of it as a flow of
information in this case sort of a
signal flow and so so this is sort of
was like an initial experiment that I
did with this which is basically here I
have a bunch of windows connected via
web RTC in this case they're all on the
same computer but they could easily have
been you know one in one country one in
a different country connected over the
web and here on the Left I'm creating an
image and I'm sending it as a video
to here and combining it with this one
and then I'm sending other videos here's
the webcam and then combining it with
what's on the bottom right and again I
would just sort of started explaining
around with these things and as I didn't
exactly know what my goals were with it
I started live coding in the console
just to sort of experiment and be able
to live change change these these things
and it's sort of like it's both a visual
experiment but also sort of a conceptual
experiment of what happens if we connect
in these ways and how can we maybe
collaborate or something so so this is
an initial experiment and then it turns
out it's not that practical all the time
if you're on the same computer to be
sending video streams
it's like takes a lot of processing
power if you're if you're just on the
same computer and so this experiment led
to a node module called RTC patch Bay
that just for me made it easy to to send
these video streams and sort of reroute
them and then I started making this sort
of in browser editor to play with this
and I'm gonna go to right now let's see
so and so this this platform is called
called Hydra that I've been working on
it lets you send video streams to other
people who are connected but it also
lets you kind of play with this way of
making images directly within the
browser and so I'm gonna actually clear
the canvas right now and so drawing from
analog synthesizer world one of the
basic things that you can make in Hydra
is an oscillator but then when you just
write oscillator you don't actually see
an
and so then to connect it to actually
see it you have to connect it to an
output and so when a and I'm doing
control enter to execute this code and
so here I have an oscillator and I'm
connecting it to an output and I sort of
use functions to correspond to this idea
of the analog modular synthesizer and
then chaining the functions as sort of
the patch cables connecting the
different pieces of of the analog
modular synthesizer and then the
parameters here can correspond sort of
like turning the knobs of of the analog
synthesizer where I can change change
certain parameters and so here here I'm
I'm changing the frequency of the
oscillator how fast it's moving and how
separated the colors are and then I can
add more more chained functions onto
this more transformations of the signal
suits similar to the boxes and the
modular synthesizer and so I can keep
adding there's oscillator rotate and now
I could add a kaleidoscope change them
repeat the kaleidoscope and so one of
the things also for with this that's
interesting to me is it's it's sort of a
lot of the ways that we make graphics on
the computer use also different
analogies for example canvas it's like
the idea that you have a canvas and
you're drawing things onto the canvas or
using Microsoft Word or something it's
the idea that you have a document and
you're you're typing text onto it and
all of these analogies kind of create a
framework for thinking about making
things using the computer and so instead
of using either the idea of a canvas
where you're drawing shapes or using a
3d rendering thing
3G is where you have a camera and a mesh
and lights and all this stuff here and
really interested in this idea of signal
flow and using signal flow as a way to
to create images and all of this this
each function corresponds to a GLSL
shaders function and a shader is a
program that runs on the graphics card
and it basically is given a coordinate
and outputs a color and it's sort of
most basic sense and so so I'll each
each of these functions either
transforms a coordinate to a color
transforms the color or transforms a
coordinate space and then adding them
together kind of lets you experiment
with these things and of course the
order that things are in matters so like
if I put another rotate here then gonna
start rotating afterwards and I can also
instead of using an oscillator as a
source I could use for any of them for
example I could use a camera as a source
[Music]
dark but that's all of you and so I
could I can also sort of apply apply
transformations to this so let's see
yeah it's a little it's a little darker
I'm gonna switch to my camera so we will
see better there we go
apply transformations and so now we just
sort of have one source and one output
I've shown you and I could I could
execute different code to sort of switch
back between the two but we thought the
whole analog synthesizer there's all
these options for like patching things
into each other and so so in this system
in order to be able to do that I
actually have four sort of virtual
outputs with which with different names
and so I could send the oscillator
instead of to the first screen I could
send it to the second screen which in
this case has a name oh one and I'm
gonna simplify this a little here so so
here I'm sending the camera to the to
the first output zero and I'm sending
the oscillator to the second output and
then what I could do is actually blend
blend these together so I could take
zero and blend it with a one and send it
out - OH - here and so here I'm blending
the two together and if I change one of
them it changes all of them again this
is all running directly on the graphics
card with through WebGL and shaders and
and if you're familiar with Photoshop or
video editing software there's different
ways of compositing pixels together so
here I'm taking the difference between
the colors if you think of each color as
a number between zero and one RGB
channels and then you can do different
sort of arithmetic but then another
thing that I borrowed from analog
synthesizer is not just combining colors
together but it's actually using the
colors of one thing to affect the
coordinates of another thing and so here
um here it's using the oscillator to
sort of warp the initial image and this
I'm all doing this sort of in separate
function changes but I could also do it
all all sort of together and so I could
just here say I'm gonna take this and
modulate it with an oscillator I'll just
show here and I could I don't know
put another oscillator on top here and
just keep keep kind of chained chaining
all these things together and it creates
really complicated patterns really
quickly and this this is just really fun
for me and then so using different Web
API is also you can not just use the
camera as an as an input but also lots
of other things oh and this is one of
the things that dance and and said about
this Sandin image processor in the
seventies is the modules are designed to
maximize the possibility of
interconnection thereby maximizing the
number of possible modifications of the
image so that's like when I'm thinking
about the functions that are there it's
how can these functions really maximize
the possibilities of the way that you
can you can mix them together and so
this is using the screen itself as an
input these are sort of experiments that
I've done using this this idea of the
video synthesizer this is using my
Twitter feed as an input using the
desktop capture API you can use any
window as an input this is using Google
Maps as a as an input and sort of
experimenting with it and this is using
a live stream of an aquarium and a live
stream of a Time Square and then live
coding them together and and this
sort of going back to the the satellite
arts project of just sort of combining
two live things and what are what kind
of possibilities or ideas about those
two spaces or arise by putting them on
top of each other and so another thing
that when it clear this here that is
really exciting for me to experiment
what this is is video feedback or sort
of using the output of something as the
input to itself or in other words sort
of recursion and so here so here here's
the camera I'm actually gonna switch
cameras here to the other one and so so
the basics of video feedback is just
pointing a camera at its own screen and
sometimes it can create really
complicated things but when I'm holding
a camera here it's not very precise and
so I could actually just do it sort of
within the code so for example I could
here make a shape and then I could blend
it with its own output which is oa0 in
this case but because it's blending the
same thing with the same thing it just
ends up also being a triangle but if I
blend it with itself and then repeat the
entire thing add a shape on top to the
repeated output it gets more interesting
and so we get sort of a fractal which is
a form that repeats itself a different
scale
and so just using this idea of a video
feedback and in pretty simple code then
you get something that's as complex
account as complex as a factor and so
another example is here I could go back
to the camera and they're actually
another example oh yeah and so I could
blend my face with itself here we go and
so then I just start to get these sort
of trails but then if I use that
modulate function again and so I
actually use the colors to affect the
coordinates and so each frame update I'm
sort of using the colors that were there
to to affect the coordinates and and
because of feedback it ends up being
this sort of liquid effect and it turns
out this is kind of similar to how fluid
dynamics or exceeds textures to to talk
about pressures and velocities and so
this is exciting to me because it's sort
of the code isn't that complicated but
these things that are supposedly
computational really computationally
complicated like navier's stokes fluid
dynamics and stuff maybe we could have
sort of an intuitive understanding of
them by thinking about them in sort of a
different way and so it closed in my
slides
oh yeah you could also use like a canvas
as a as an input which is kind of fun
and yeah I've just kept playing around
with this this sort of this thing or
this is reaction diffusion made using
using feedback and this is one of the
things that dance and it said in the 70s
about the idea of the feedback loop is
put yourself in a feedback loop yes the
reason you put yourself in a feedback
loop is so you can learn something or
rather learn to do something you know
you do something in the results get
processed somehow perhaps just by
yourself just looking at it and you do
another thing in the results of that get
processed and your action can constantly
be tuned or corrected or refined by what
you see and so this has been interesting
to me because although I've built this
software from from basically scratch
like there's almost no using no
libraries and stuff but I still don't
know what's gonna happen and having the
editor there and being able to change it
in real time and see the results is sort
of really resonates with me both as an
artistic process as well as a process
for developing code and just sort of
investigation as well and so also one of
the things they've done is made a
Twitter bot where you can you can post a
sketch to the Twitter ah and there's
actually an anyone who's different
people who've been using the editor can
can post something here and then if you
actually click click on on it in the
Twitter bot it directly opens the code
and so then you can edit directly edit
the code here and so here I'm editing
this I got a little fast and then I can
upload it again and here let's say J s
conf Asia
and then upload it and we'll see if it
shows up here but yep that goes so it
and so it shows us what we just did but
it also shows the sketch that it was
sort of based on here and that this one
was based on one earlier and it's
thinking about like oh we're all just
kind of exploring the possibilities of
this the game almost out of time yeah
but I wanted to just quickly mention
this idea of live coding so I've started
I live code visuals with this so
sometimes in clubs or in different
events with experimental music and I
make a write code live to make visuals
and this is there's another people all
over the world doing this using
different languages and sort of
experimenting with ways of programming
and sort of showing the code while there
while they're performing in this there's
this language called tidal cycles that's
for making music and as somewhat similar
ideas as Hydra for making visuals in
that although it's in in Haskell but
everything that you write in title is
sort of a transformation of a pattern in
time and using that you can make music
so yeah we have a live coding group in
Bogota there's little groups all over
the world there's one in India and I'm
meeting with some people in Indonesia
next week insects
this is last week in Bogota using title
to make music
Haeju to make business anyways thank you
so much
[Applause]