[Music]
I always forget
[Music]
I'm gonna do this this dot this dot this
star this star song never forget the
bits taught somebody composed that song
for me hello and welcome to a very
special summer edition of the coding
train so I'm kind of on a little bit of
a hiatus right now you might have heard
me talk about where this very light
content happening in July and August
coming back full steam full force in
September um so I want to talk a little
bit about schedule and stuff and what's
happening and what you can expect but
before I do any of that but in order for
you to follow along with the main
content that is the content of today's
livestream which is a very special
exciting content that very excited about
you do need to download some software so
I'm just going to tell you what to
download spigen said it going in the
background while I babble babble on and
on and on about other stuff so I'm gonna
open up a web browser something like
this go go glue googly I don't know how
to pronounce that chromic home eh I
think it's pronounced web browser and
I'm going to ask you to navigate to if
you so choose runway and al.com so go to
runway ml comm then you'll want to head
over to this link under download beta
and here it should start downloading
automatically and there are versions for
Mac OS Windows Linux anything else so
whatever you're out by the way I'm
talking to somebody over here normally
I'm just in this room talking to myself
but I am talking to Kris
Valenzuela how'd I do on the
pronunciation pretty good one of the
founders and creators of Runway who is
here in the chat say hi to Chris in the
chat to answer your questions and to
answer all my questions as I fumble my
way through trying to learn about Runway
so I'm gonna come back to talking about
what Runway is and what I'm planning to
do in today's livestream but I just
wanted to at least set you on - Dan
loading this if you want to and I'm
going to give you a coupon code
a little bit to get some extra credits
as well so but before I do that so while
you're downloading that what's happening
here where are you where am I
where are we with is the what is that
what's what is the meaning of the
universe that is not a question I will
answer so this is a little neighborhood
YouTube channel called the coding train
and every time I say coding train I then
have to blow a train whistle it's in
this contract I signed awhile ago and
where I make computer programming
tutorials and otherwise wastes reading
things like random numbers and most of
my activity happens during the school
year because I also work at a school New
York University I teach in a program
called ITP
and also a that's a graduate program an
undergraduate program called ima they're
both part of Tisch School of the Arts so
summer is a little bit lighter for me in
terms of content anyway but this summer
is a particularly unique summer in that
this room that I'm standing in right now
is being closed down and I'm moving the
cameras and equipment to a new room on
our in the Brooklyn campus sometime this
month or next month so I mean I'm gonna
lose some time where I don't have a
recording studio and we'll be setting up
a new recording studio which hopefully
some better stuff and back full force in
September however there are many videos
still outstanding that I am working on
editing together and releasing number
one is and this is very exciting a full
documentation of a video about all of
the equipment and software in this room
that I'm using so that's probably gonna
come out in the next couple weeks which
is good timing considering that I'm
closing this down so this coding train
studio which has been here since I don't
know when I first started I got a look
back and figure out what was the first
livestream I did from this room
the first livestream on the coding train
I think was 2015 or perhaps earlier than
that but I was I was doing everything
from my office downstairs for a while
anyway that's not super important so
that's coming out I have a two-part
series on how to program your own
processing java library which I know
everybody on the Internet
just waiting that's gonna blow up
YouTube that's gonna be like highest
viewed video ever where I show you how
to build a Java library in Eclipse cuz
that's like all the rage now that's
whatever is doing right right okay take
a deep breath here for a second that's
coming out of an the edited version of
the tic-tac-toe coding challenge that I
did on a previous livestream is coming
and also if I can get my act together
because I forgot to record some stuff
for that today a edited version of the
neuro evolution steering car racing
coding challenge thing so that's also
coming so there's a bunch of content
that will be coming out in the channel
but this might be the last livestream
until September or I might pop back for
a surprise livestream like this one on
subtopic leg runway okay now I'm looking
at the chat now let's say you happen to
be supporting the coding train as a
member on YouTube or a patron on patreon
and I think I am going to I'm putting
this out there the University I'm not
sure about it but I might shut down the
patreon and move over entirely to
youtube memberships you know I hesitate
to have everything all in one platform
but there are some advantages to that
and so stay tuned to information about
that but you might be wondering when I
signed up for a membership and suddenly
now there's no content for two months
well I'd suggest that you unsigned up if
that's really an issue but one of the
things that I will be head-down working
on for the next six weeks is a dish 2nd
edition of the nature of code book and
I've talked about that in previous live
streams so I might do some member-only
live streams with work sessions on that
book and share some PDF so that's that's
what I'll be doing in terms of the
member community in the next couple
months I'm looking over at the chat that
I think that's the main sort of
announcement to eat things that I wanted
to talk about nature of code books
summer schedule tell you to download
runway I don't know anything else
take a momentary break here I know what
it means to take a break I mean I'm just
gonna take a break from talking anybody
have any trouble downloading runway if
so let us know in the chat
[Music]
now I will tell you that coding tree is
sponsored by water it's not exactly wet
and there are different kinds of flow
like laminar flow and turbulent flow I
think what I'm drinking it's more of a
turbulent flow refreshing actually I
make that's my joke
it's very lame but it is my joke um this
there's no sponsor for today's
livestream however I do want to mention
that I am actually an advisor to the
company runway that makes runway so this
well this isn't exactly like a paid
product endorsement there is that
relationship there which I feel is
important to disclose so this is
something like of a sponsored video in a
way by a runway itself and you can find
out more about the company runway itself
and the people behind runway and I know
they're hot doing a lot of hiring so
maybe you want to apply for a job with
runway I'm all of the runway website to
see if there's any questions that
anybody had any cloth Vogel rights
I have runway running wax or 82 thank
you for your membership of the coda
train look you have a little comma
waving emoji icon thing I forgot what
that means a couple a couple couple of
two months six months one year who's who
in the chat has the longest I think if I
go here to YouTube / the coding train
/join it'll tell me does that tell me
anything no maybe because I'm already a
member oh no because I'm not logged in
who knows well look I'm live okay
there's me alright so it's okay cookie
crumb is asking is runway okay for
beginners because I'm a beginner yes
why said that way too loudly let me said
we're Conley yes so I think there are on
the runways a really interesting thing
and I think that it would be best for me
to start using it to explain it but
because I think there are aspects of it
the the whole system by its nature the
way that it's built is a piece of saw
or is meant for beginners but it is a
tool that allows you to explore pretty
deeply the world of machine learning
models some of which are quite
sophisticated and you can wander down a
lot of advanced and tricky and complex
rabbit holes so I think there's a wide
range here of kind of understanding and
it's required not required but that that
is part of using runway but what I like
to do on the coding train is have
everything I'm making my assumption that
you've never seen it before I would say
the only Priya lis not any prerequisites
for today which is pretty rare but
though eventually at some point I'm
gonna move towards using runway with
maybe processing and with p5.js and then
you might the prerequisite would have
some familiarity with those platforms
and basic coding knowledge but I am here
to help I am here to help and yes this
is appropriate for beginners all right
so I think oh somebody gave me a super
chat which I have to acknowledge Charlie
England thank you thanks for that all
right
any update on our mysterious bug that we
were talking about earlier it's fixed Oh
exciting I was all exciting to debug say
oh we have a new member welcome to new
member ball Simpson thank you for
joining the coding training ring make
all the noise things I like you know I
should automate all this stuff whatever
okay so let's get let's get going so
what I'm going to do so I this is this
is something slightly different from my
usual live streams in the sense that
this is entirely themed around runway
and ultimately it may not make sense or
be necessary to edit this into smaller
shorter videos but just in case it does
happen to make sense to do that
because there's a lot of me babbling and
I'm sure I'll waste some time later
chasing down some weird buggers piece of
documentation or reading random numbers
I'm going to assume actually that I'm
going to make two video tutorials
extemporaneously improvising but one
will just be in
reduction to runway as a platform and
signing up for a runway account and then
the other will be maybe doing a coding
challenge where I try to get the results
the output of a machine learning model
into a p5.js sketch I'll get to that
later
and really again let me just mention if
you have questions about runway ask them
in the chat because you can have a whole
completely separate conversation without
me with Chris one of the founders and
creators of runway here and maybe
they're also even some other people from
runway watching hi people from runway
I'm probably gonna screw this up so
careful what you wish for
alright here we go alright and let me
just cover one other thing because how
around is asking do you have to pay to
keep using it so I will get into this as
I look at runway runway is something
that you can use you don't need to enter
a credit card you can use it without
paying any money however there are some
caveats to this the one of the things
that I'll be doing is running the
machine learning models with runway in
the cloud and that requires paying a fee
by the minute or by the hour by the
minute for cloud GPU usage every runway
user when you sign up you automatically
get $10 of free credits and as I go
through the signup process I'm gonna
give you a coupon code it's just coding
train which will give you an additional
$10 so you'll be able to get $20 in
credits for free and you actually do
quite a bit $20 in credits you'd be
surprised and then towards the end of
the livestream maybe I'll talk about
ways that you can actually run the
machine learning models locally on your
laptop if your laptop has a GPU
obviously there's some limit there's
quite a bit of limitation to that
depending on what computer you're using
but that would also allow you to run it
without paying any money whatsoever I
get that pretty much right all right
okay so um like you without the beard
but I'm wearing the beard where is that
a thing you say wearing the beard that's
weird I don't think you wear a beard
maybe you do whatever alright so let's
go to the homepage of runway and I am
now going to start artificially as if
I'm starting over introducing you to
runway and then going through the steps
to download and sign up for runway and
we'll see how that goes and I think what
I'll do you I think in the new studio by
the way I won't have this turning the
cameras on and off thing which I'm
actually quite sad about because it
brings me a lot of joy Eli right no the
beard wears you which i think is a very
profound statement okay by the way I
just have to highlight this look see
this is the homepage of runway M L can
zoom in here we see this is some sort of
computing machine running the runway
software this is probably like the dense
pose model maybe and then it but I just
want to like highlight this go over here
into the corner here oh look at this
nature of code by some strange person
named Daniel okay there we go nice to
see that there there's some other books
there's a lot of life's little Easter
eggs in here if you keep looking here
comes everybody this is a great book by
Clay Shirky I think I might have read
all of these books here we'll see
all right great alka reports that
signing up and entering coding train
code went smoothly which is great all
right how do I have no plans for this I
mean everybody's aware of that but I
just feel the need to say that one more
time since Christmas hello and welcome
to a tutorial series on the coding train
about a piece of software called runway
so what I'm gonna do in this particular
video is just show you the runway
website I'm going to download the runway
software I'm gonna sign up for a runway
account and if I kind of click through
and show you the basics of how it works
now what is even runway and why might
you use it nope no before I even get to
that though I feel like it's important
to say a couple things one is runway is
not something that I've made runway is
made by a company also called runway
they happen to be some of the founders
of runway and creators of runway or
alumni of a program where I teach that
make sense ITP and I'm an advisor to the
company but this is a tool that you can
use
starting today if you download it for
free with some with a coupon from the
coding train let me do this one again
I'm just I I won't keep doing this over
and over again but I just needed a
little like warm-up that was a warm-up I
said too much I don't need to go on and
on about this way too much
okay hello and welcome to a new tutorial
series on the coding train using a piece
of about a piece of software are called
runway so what is runway how do you
download and install runway and kind of
tinker around with it that's all I'm
gonna do in this particular video now
let me clear runway is not something
that I've made a runway is made by a
company a new company called runway
itself it's a piece of software you can
use you can download it for free you can
use it for free there are aspects of it
that require cloud GPU credits which
I'll get into later and you can get some
free credits and a coupon code that
you'll find in the description of this
video
but really I want to just talk to you
about what is kind of so excited about
it I'm planning to use it in the future
in a lot of future tutorials and coding
challenges and teaching things that I'm
gonna do and I also should just mention
that I am an advisor to the company
runway itself so I'm involved in that
capacity all right so what is runway now
right here it says machine learning for
creators bring the power of artificial
intelligence to your creative project
with an intuitive and simple visual
interface start exploring new ways of
creating today so this this to me is
like the core of runway I am somebody
who's a creative coder I'm working with
processing and p5.js you might be
working with other pieces of software
that's just commercial software coding
environments you're writing your own
software and you want to make use of
recent advances in machine learning
you've read about this model you saw
this YouTube video about this model can
you use it in your thing well before
runway one of the things you might have
done is find your way to some github
repo that had like this very long readme
of about all the different like
dependencies you need to install and
configure and then you've got to
download this and install this and then
make build this library and you can
really get stuck there for a long time
so runway is an all-in-one piece of
software with an interface that
basically will run machine learning
models for you install and configure
them
that you having to do any other work but
press a button called install and it
gives you an interface to play with
those models experiment with those
models and then broadcast the results of
those models to some other piece of
software there's a variety of ways you
can do that broadcasting through HTTP
requests through OSC messages and all
these things might not make sense to you
which is totally fine I'm going to poke
through them and show you how they work
with an eye towards at least showing you
how to pair a runway with processing and
how to pair a runway with p5.js and I'll
also show you where there's lots of
other examples and things you can do
with other platforms and stuff like that
all right that was pretty good alright
so I'm going to so the first step you
should do is click here under down the
firm so the first step you should do is
click here on your download runway beta
it will automatically trigger a download
for Mac OS Windows or Linux I've
actually already downloaded and
installed runway so I'm gonna kind of
skip that but skip that step and just
actually now run the software so I'm
gonna go over here and I have I have the
runway icon already here as part of my
what does this thing close the Mac the
doc knows I'm going to click that run
run right ah and now it's saying welcome
to runway sign-in to get started ok so
if you already have an account you could
just sign in with your account I do
already have an account but I'm going to
create a new one just so we can follow
along with the process so I'm gonna go
here create an account I'm going to
enter my email address which is Daniel
at the coding train comm then I'm gonna
make a username and password the user
name will be choo-choo do you think
that's gonna be taken already my
password alright let's let's pause for a
second generate a password not currently
sponsoring today's livestream but hey -
Lane I think you can sign up at - like
on flash go to trade or something
they were sponsor of a previous
livestream can find that there there a
password generator somewhere in here
how my password changer tools password
history password generator there it is
any requirements of the password I
should know about let's do like oh wait
that's a good idea
alright hold on everybody that's it okay
I'm going to generate a new password oh
that you can't see it a copy password
I'm going to copy it I'm copying the
password in and I'm just gonna also
please hold I kind of put you back here
there we go alright now that I've put in
my very strong password I'm going to
click Next and I'm gonna give my details
Daniel
Schiffman the coding train create
account
ah and it's giving me a verification
code to Daniel at the coding train comm
let me grab my mobile device that'll be
the easiest way for me to read that
email oh it's even got a notification
google thinks that an email from Runway
is a very important email cuz I have it
to set up to only give me a notification
for important emails I'm gonna say six I
don't think there's any issue with me
I guess maybe though I will shut this
off as well just just for security sake
if you can like determine what keys I
pressed based on audio signals then
you're doing very well
validate okay put you back on their
account has now been creat account has
now been created and I can click start
alright so this is now I'm just gonna
like minimize the browser so we can be
clear on what we're actually seeing
that's part of runway what's this year
Oh
hold on a sec let me turn that off is
that the password that I just revealed
by accident this is going remarkably
well that was totally the past who knows
my runway password account now all right
I'm gonna change the password I'm sure I
can do that manage my account edit
profile a password change password
current password new password beat you
all hackers are everywhere update great
I changed my password so now my account
is secured again I've logged in and we
go back to a runway it logged me out
that was nice
sorry about that everybody okay close
this there we go close the password
generator Oh looks like I got some sort
of super chat which I will now check on
thank you
Oh XR 82 that's very nice of you just to
say thank you okay there we go all right
this is everything you imagined the code
exchange could be just me trying to like
not give people my password by accident
that's basically what 90% of the time is
all right all right so once you download
it install Broadway and signed up for an
account logged into your account you
will find this screen so if you've been
using runway for a while you might then
end up here clicking on open workspaces
because workspaces are a way of
collecting a bunch of different models
that you want to use for a particular
project into a workspace but we haven't
done any of that so the first thing that
I'm gonna do is just click on browse
models and if you've never run run way
and clicked on browse models this is
gonna be a very exciting moment for you
because there are a lot of models which
you suddenly can play at your fingertips
that you could just play around with and
I haven't even planned this so I think
we should just pick one somewhat
randomly I'm going to use style gam
later
anybody have one that they see that they
want to request I would love to use
dense pose would be kind of fun
attention Gann oh what's the one is that
one where I could draw oh yeah is that
spayed Koko yeah so I think that would
be a good one
Thank You sundar ten all right so so the
first thing that I might suggest that
you do is just click on a model and see
what you can do to play with it in the
runway interface itself because one
thing that's really wonderful about
runway is as a piece of software in an
interface you can explore and experiment
with the models to understand how it
works what it does well what it doesn't
well do well what it does it all before
starting to bring it into your own
software or your own project so I'm
gonna pick this spayed Coco model which
I have never looked at before it's very
legitimate me I have no idea what's
gonna happen when I click on that and
now here I can find out some more
information about the model so I can
find out what is the model do it
generates realistic images from sketches
and doodles I can find out more
information about the model for example
this is the paper that describes this
model semantic image synthesis with
spatially adaptive normalizations
trained on cocoa stuff dataset remember
when someone asked is this big is this
is this tutorial for beginners well is
for beginners in that you're a beginner
you can come here and play around with
it
but there's there's a lot of it you can
go very deep too if you want to find the
paper read through the notes and
understand more about what this model
how it was built
what data it was trained on which is
always a very important question to ask
whenever you're using a machine learning
model so I can also we can see there are
attributions here so this is the
organization that train the model these
are the authors of the paper we can see
wet what the size of it
when it was created if it's CPU and GPU
supported and then very important it's
down here but I'm actually going to
highlight it here we can click on the
license so this is Creative Commons
Attribution non-commercial share-alike
4.0 license so I'm not gonna pretend to
be a lawyer here I'll be able to give
you proper advice on the you know the
beyond the scope of what I'm doing here
is the nuances of what all these
licenses mean and how where you can use
them but you can find out and do that
research yourself right here in the
runway interface we can also go under
gallery and we can see just some images
that have been created so we can get an
idea this is something this is a model
that's themed around something called
image segmentation so I have an image
over here what does it mean to do image
segmentation well this image is
segmented divided into a bunch of
different segments those segments are
noted by color so there's like a purple
segment a pink segment a light green
segment and those colors are tied to
labels in the model essentially that
know about a kind of thing that it could
draw in that area so you could do image
segmentation in two ways I could
an existing image like an image of me
and try to say like oh this I'm gonna
segment it this is where my head is this
is where my hand is this is where my
hand is
or I could generate images by sort of
drawing on a blank image saying put a
hand over here put a head over here and
it looks like at least the example we're
seeing here is kind of like a what do
you call those things it's an art not a
landscape yeah that's a bowl of fruit
that has a name how come I can't think
of that word so chats gonna say it
instantly art bowl of fruit okay I can't
think of this word I'm going crazy
still life it's not what I mean I think
it's a still life it's not what I mean
maybe it's just still life I thought it
was something else
whatever all right and then we'll get
editors who version of this video
okay so that's what a big segmentation
is at least in the way that I understand
it okay so let me go here now what I
want to do is I want to add this too so
now I want to use this one so what have
I done so far thank you James Craig and
all these super chats today this is not
usually happen it's YouTube like
promoting super chats thank you James
Craig I really appreciate it you that
gets a train whistle okay oh what a time
to be alive all right where was I what
have I done so far I've downloaded
runway I poked around the models I've
just clicked on one now I want to use
that model I want to play with it I want
to see it run so I'm gonna go here to
add to workspace it's right up here add
to workspace now I don't have a
workspace yet so I need to make one and
I'm gonna call this workspace red I just
want to leave it as this name red
current multiculturalism so it'll
automatically name a workspace for you
and did it did it breaking news I'm
being told that these workspaces are
named from colors and art isms fruits Oh
fruits fruits and art isms so maybe you
got blueberry abstract expressionism for
some
all right so I'm a bit creepy I don't
know I'm gonna name it though I'm gonna
say coding trained livestream so I'm
gonna do that when hit create now I have
a workspace and you can see this is my
workspace
I have only one model added to this
workspace over here and it's kind of
highlighting up for me right now what to
do I need to choose an input source so
every machine learning model is
different some of them expect text input
some of them expect image input some of
them might expect input that's from it's
arbitrary
scientific data from a spreadsheet then
the model is going to take that input in
run it through the model and produce an
output and that output might be numbers
or it also might be an image or it might
be more text so now we're in the sort of
the space of a case by case basis but if
I understand image segmentation
correctly I'm pretty sure the input is
going to be both an image and the input
and the output are both going to be an
image right so let's make a little
diagram here this whiteboard has shut
off but let's make a little diagram so
we have this what was this model called
again spade spade cocoa so we have this
machine learning model presumably
there's some neural network architecture
in here maybe it has some convolutional
layers this is something we would want
to read that paper to find out more
runaways can allow us to just use it out
of the box and you know it certainly
would always recommend reading more
about it I want to learn more about how
to use it so my assumption here is I'm
in my software that I want to build I
want to I'll maybe create a drawing
piece of software that allows a user to
segment out an image so you could
imagine maybe I'm gonna like kind of
draw something that's one color look I
could use different color markers
like I'm going to fill you know I'm
gonna sort of fill this image in with a
bunch of different colors one of these
markers don't work this is not important
and then I'm going to feed that into the
model and out will come an image so we
have input and we have output and again
this is going to be different for every
model that we might pick in runway
although there's a lot of conventions a
lot of the models expect images as input
and output images some of them expect
text as input and output an image or
images input and output text etc and so
on and so forth so here under choose
input source I'm gonna click
segmentation oh wow I'm not even the
right thing shoot well then let me um
can I get back to where I was before let
me actually just remove it well actually
if I click the file this is fine I don't
know why I'm obsessed I'm gonna try I'm
gonna delete it just so by the way one
thing you can do is you can delete a
model but I'm gonna delete it and then
I'm just gonna add it again in case you
missed that part you get to see it again
I'm gonna add to workspace cucumber
multiculturalism no I don't want to new
I want to go to my workspace Oh add a
model so what did I what did I miss here
let's see I'm discovering the runway
interface so I click here recently used
Spade cocoa add to workspace there we go
oh oh you know it must sound uh saves my
workspace because I there was nothing in
it
alright no problem we're gonna do it
again red the nice thing about this is
we get to have more automatic workspace
needs let's see so I'm gonna say coding
trained live stream again I just wanted
it to be I just want it to be back to
here in case we edit out I guess I can't
not do it without thinking about we're
gonna edit it alright and so now what I
want to do is choose the input source in
runway for the model so something that's
going to produce a segmented image and
so that could be coming from a file it
could actually come from a network
connection which I'll get into maybe in
future video or you can explore on your
own I'm just gonna pick segmentation
amount this is like the greatest thing
ever
because what just happened is runway
your image segmentation is a common
enough feature of machine learning
models that runway has built into it an
entire drawing engine so that you can
play around with image segmentation so
and you can see these are the colors for
different labels so maybe what I want it
looks like it's a lot of transportation
stuff so maybe what I want is let's try
slette stride wrong some people so I'm
gonna do that if we do it edited version
this will get sped up how am i doing
let's draw two people I don't think this
is actually how people look in a real
image look good it'd be interesting to
see what it does with this look and
let's put an airplane flying above I
wonder if I don't need to do I need to
be interesting I think about 20 does it
matter if I do it in the shape of the
airplane or is it just gonna figure it
out because it knows what the shape of
an airplane is that's actually a much
better airplane than I imagined myself
drawing what else elephant ooh oh boy
there's a lot other stuff that's not
transportation wine glass this is gonna
be a very large wine glass well two
people with a airplane and a wine glass
flying overhead okay oh now I'm gonna
choose an output and I just want to do
preview right cuz preview right now is
like I'm not actually I don't need to
export this I don't you need to use it
somewhere else I just want to play
around with it in runway itself so I'm
gonna hit preview and ah and here comes
the most nothing and now here's the
thing so okay sorry now I have selected
my input which is just the segmentation
interface of runway itself I have
selected my output which is just a
preview now it's time for me to run the
model and here we go run remotely so a
remote GPU enabled and you can see just
by signing up for
run away I have $10 in remote GPU
credits I be interesting to see how much
just running this once actually uses so
one thing I mentioned now if you want to
get additional credits I can go over
here this is like the sort of icon for
my profile I can click on this I'm going
to go now to here get more credits is
that where I want to go yeah I'm gonna
go to get more credits and this is gonna
take me to a browser page and I could
have certainly pay for more credits but
I'm gonna click here and I'm going to
redeem credits by saying coding train
right here so if you would like to get
an additional $10 in credits you can do
this
and we can see now I should have 20
dollars in credits so I can close that
minimize the browser and go back to here
my workspace my model whoops where do I
want to go this is this is workspace
here we go there we go I'm back here so
this icon up here just so we're clear
this icon up here is your workspaces of
which I only have one with one model
that's connected to remote GPU and if I
wanted to look at other models I would
go here to this icon alright now I'm
gonna press run remotely
[Music]
couldn't have a very long sound effect
is so beautiful I cannot believe it so
spade cocoa machine learning while
generating it's really interesting to
see the results here so you can think me
knowing nothing about this model kind of
how it works and what to expect get some
pretty weird results with it probably if
I were a bit more thoughtful maybe if I
even like filled in the entire space
right I probably I left so much of it
blank I also included like a giant wine
glass with two people it's very kind of
creepy looking although this I think
this sort of resembles me in some
strange sort of way and we can see here
look at this five cents here's my five
softimage but what's that oh I could
live paint okay and apparently I can
live paint
all right that's crazy let's try doing
that so the reason why I took a long
time was it was just kind of booting it
up so one thing I should mention is the
reason why that took a long time it was
like spinning up the server and
everything to start actually running the
model but now that it's running in real
time it can happen much more quickly so
let's try adding I want to add something
like but what if I want to add an
umbrella let's add an umbrella to this
person and there we go and like this one
this person's gonna have a suitcase
whoops don't click on that oh and I'm
drawing down here look at me like not
knowing how computers work this is the
this is the inputs I need to draw here
let's draw a little suitcase what else
so let's put on some eyeglasses does to
make this person look more like me I
keep doing that it's funny how my
instinct is to do that put on some
eyeglasses here those
great high glass tennis racket cop nice
Bowl I kind of want to start over
let me start over how do I can I clear
this like they'll probably this Oh fill
bucket oh that's what it been that's
useful is there a clear the whole thing
button I can undo or trash there real
clear so let's try filling it so what
would be a good thing to fill it with
that's like background dirt fence
let's try floor wood so let's try
filling it with a wood floor oh whoa
then let's put let's try to put like
some fruit so was there like apple here
so let me put an apple gonna paint that
this is looking much better now let's
put like an orange right next to it
let's put a couple oranges and make a
little bowl of fruit Wow
crazy Wow where's my search oh here it
is let's see what else what other fruits
what is there a banana it's probably
banana okay so let's put a banana okay
this I gotta stop that's pretty amazing
so again here was just a little moment
later of being being a little more
thoughtful to think about how this model
actually works and what what might
happen you know I if I knew if I looked
at the data set which is the fairly well
known I imagine cocoa image data set
that's probably going to be give me even
more information to think about what
it's going to do well but you can see
how it's able to sort of think about a
sort of little seat a little little pile
of fruit here on a you know wood
background almost looks a lot more like
cloth like it's sitting on a table
pretty realistic and yes Charlie England
points out which is correct this is
continuing to use the GPU credits and we
can see that you know still though I've
even with doing a bunch of live painting
I've just used ten
since there so you could do a lot with
the free ten dollars just in playing
around
Simon Says Spade cocoa I'm just curious
here to Google Spade cocoa dataset or
maybe that was from like it's the cocoa
dataset yeah
cocoa dataset is this data set so I
would encourage you also to check out
this URL but with a lot more information
but something about the data that was
used to Train this particularly in the
segmentation how all right okay let me
take a minute here let's see if people
have any let me take a minute to see if
there's any questions that people want
to ask in the chat that I can either
answer myself or asked Chris about yes
and so I noticed that Chris just wrote
in the chat some models only work on
remote GPU so if you won I don't know if
this do you know if offhand if this is
one that will work locally no so this is
one that only works on remote GPU other
models work locally I think I'm gonna
try to run pose net which can run
locally and so it won't use any GPU
credits for that and then some models
will work locally but require another
installation and another dependency
which is something called docker which I
could talk about or install at some
point but I know that runway is working
on making that process even easier to be
able to run stuff locally okay can I
lower resolution to speed up that's an
interesting question so I I can see here
that the resolution is set to 640 by 360
is that something that can change with
this model
no but the but the model I couldn't
change that in the interface yeah so so
this is a good question because you know
the way most machine learning models
work in particular if they're fixed
inputs and fixed outputs not like a kind
of sequential model the output is fixed
the input is fixed it requires a
particular resolution because the neural
network is expecting a certain amount of
numbers to come in so you can do to some
extent you could do resizing and
resampling if I go back to my diagram
here like I could do some resizing and
resampling here or I could do some
resizing and resampling here before to
prepare the input or after I get the
output but the model itself the input
image has to be a particular dimension
and the output is only going to come in
that dimension some some models are
released I think with you know like sort
of different versions of it with which
may be output different dimensions I
think I've seen that but in this
particular one it's fixed dimensions
okay all right I I could like play with
this forever loop I could play with this
forever but I think I'm gonna not all
right all right so now let's let me
think about this what else do I want to
show here before I move on I think what
I'll do is to separate like sort of like
tutorials one using pose net with
processing and then one using this style
gain a rainbow's with p5.js but I want
to think it's just in sort of this like
general tutorial about the platform as a
whole what else do I want to say so
let's look at so input wise I could also
so I chose to do segmentation here but I
could also use a file so if I wanted to
open a file on the computer I could do
it that way and then output if I change
to export I could also actually export
that to a variety of different formats
but of course I could also write here
just under preview I can click this
download Save button
and now I am saving forever more that
this particular image is a file now
what's really important here actually
more important here is under network so
if what I wanted to do is click over
here under network this means I can now
communicate with this particular machine
learning model from my own software
whether that's software that I've
downloaded or purchased that somebody
else has made that speaks one of these
particular protocols or my own software
that I'm writing in just about any
programming language or environment if
you if you have a framework or module or
a library or support these types of
protocols so and the one the nice things
here if I click on JavaScript it's
actually we can see there's actually a
bit of code here that you can actually
just copy paste into your JavaScript to
run it directly so I'm gonna come back
oscy is also a really popular protocol
messaging Network messaging protocol for
creative coders stands for open sound
control allows you to send data between
applications so I will also kind of come
back and in a separate video and show
you about how some of these work
I should also probably mention that the
runway software itself works in a very
similar way to a piece of software
called Weka Nader that you might be
familiar with Weka Nader is a software
that was created by Rebecca fee brink
years ago that allows you to train a
neural network with data sent over OSC
messaging and then get the results of
that after the fact the real sort of key
difference here is runway is really set
up to support a huge treasure trove of
pre-trained models whereas Weka Nader
was more for training on training neural
networks on the fly with small bits of
data but and I will say that one of the
things that runway is planning maybe by
September is to start coming out with
features for training your own model as
well okay so people are asking I noticed
this is being discussed but I want to
see if I understand this correctly as
well so the model is still running
but unless it's not actively so it's
still you right now it's using cloud
credits because it's running yeah but if
I were to click so I in a way I like
wasted a little bit of my credits by
talking without using it but if I wanted
to be but as soon as I click stop it's
no longer running on the cloud
ok ok I'm just looking at the questions
alright I think that's most of let me
wrap do a little wrap up here in case we
try to like edit this together as
something that's ooh chain ER coming
soon that's exciting so if I go here I
just want to understand the sign I do
this ok got it
ok alright think I guess yeah let me let
me let me wrap this up here and then
we'll go to doing what time is it I
plenty of time go to do a pose net with
processing cuz I think that'll be a good
example of seeing the full workflow so
thanks for this watching this
introduction to runway just sort of the
basics of downloading installing the
software what it is from a high-level
point of view um what the features of
the interface work how to get some free
cloud credits and what I would suggest
that you do after watching this video is
download run it run the software and go
to this browse models page so you can
see there's a lot of different models
for looking at motion generative
community text recognition cook around
here let's try this recognition 1 face
recognition dense cap where's pose net
in here that might be under motion dense
pose pose net so here's a model called
pose net which performs real-time
skeletal tracking of one or more people
I've covered this model in other
libraries like the ml 5 jazz library
with tensorflow J s and so what I'm
gonna do in the next video is use this
model pose net in runway with my webcam
running it locally on this computer
without requiring cloud credits and then
get the results of this model in
processing it's
so I'm gonna show you that whole
workflow but poke around click around
find and find a model you lock like poke
around click around find a model that
you like let me know about it in the
comments share images that you made with
it and look forward to seeing what you
make with runway ok thanks for watching
all right I'm just looking at the chat
but I guess I don't really need to ok so
next what I'm gonna do is kind of I'm
gonna go to let's let's find something
so I want to go to runway ml processing
I'm gonna actually just stay here so I
need this I want to have pose net up
here and running okay so I'm gonna go
here I want to make sure I have
processing up so let me run processing
oh wait a sec here
[Music]
I'm just getting set up for the next
sort of tutorial module here look let me
just peek at the example to see whether
I want to try to just sort of like write
it or just copy it
so you need the runway hose before OSHA
requires the OSC library there's a lot
of stuff going on here I think what I'll
do is I'll bring this stuff in slowly
piece by piece
[Music]
I've got runway open
I've got processing open and I've got
the browser open a V in the chat a is
asking what about creative coding in
blender using Python scripting interface
to generate geometry or something so
yeah you can you know I'm not an expert
on blender I've used it almost you know
I've used it for maybe less than five
minutes of my entire life but most
pieces of software and I'm pretty sure
blender included come with capabilities
that allow you to communicate with the
blender software itself over the network
a blender is a special case because it's
completely free and open-source software
so if it didn't have that capability you
could actually build that capability in
or more likely somebody has already done
this I know there's like add-ons and
thanks for blender but absolutely if you
want it to generate I would say like
textures for a 3d model in blender if
you wanted to control puppeteer a like a
character in blender with a like pose
detecting model these are all things you
could absolutely do by pairing blender
and run way together all right so let me
this I might also need to do some
diagramming here to talk about the
different pieces actually there's no
clap I don't think there's going to be a
cloud involved here I'm correct all
right
so the next the next thing that I'm
gonna do I will get started with right
now okay just click this here let me
just check a couple things and make sure
things are working yep all right excuse
me hello and welcome to another video
tutorial about working with Runway and
running machine learning models in
Runway itself now before you watch this
video tutorial if you've never used
Runway before you might want to go back
and look at my introduction to runway
how to download and install it but to be
honest you probably could figure that
out if you just go to runway ml dot-com
and click on download beta you're gonna
want to then download and open the
runway software you'll go then to browse
models
I might go here under motion and I'm
going to click pose net and you'll find
yourself right here so this is where I
am I've installed runway I've downloaded
it and I'm on the page in the runway
software for the pose net machine
learning model now what is pose net pose
net is a machine learning model that
performs real-time skeletal tracking of
one or more people and guess what I'm a
person and I've got it so let's actually
first click Add to workspace so I
already have a workspace that I've made
in the previous video called coding
train livestream I want to choose an
input source which I want it to be my
webcam so yep run away go right ahead
and there I am and then I want to choose
an output source which eventually I want
to be processing because I want to get
the results of running this machine
learning model pose net into processing
itself but for right now I'm just gonna
click on preview so I click on preview
oh and have to run but guess what so
this is different than what I've showed
in previous videos I've got an option
for run locally and in fact this model
can only be run locally it would be
silly to run this one in the cloud
because I'd have to spend all this time
sending the data over the network and
it's very easy for it to
this is a very small fast model can be
run on most modern computers so I'm
gonna click run locally so it requires
no GPU credits absolutely can be used
for free and we can see there it goes
it's running right now it is making
guesses as to where the various key
points of my skeleton are on my body in
the output that's view below I'm just
looking at this here so I'm looking at
now one of the nice things that I'm
seeing here is there's so many options
that I can like tweak and I can tweak
all these in real time right now is
there any of these that would make it I
think if I do single pose it'll run
faster right oh but I have to stop
running it to change those but these can
I can change in real time I see this
welcome to the cony train where a person
on the internet makes tutorials about
things that he's never used before but
that's the way to learn yeah just try it
you just have to just go for it leap of
faith here alright so let me just
actually stop for a second let me change
this to single pose can we do run
locally again
all right that's good to know okay so
then are there other things here I'm
just curious that might that that might
speed up the the performance
architecture like if I make it lower so
let's see architecture ah okay so we can
see this option here mobile net
architecture larger the value the larger
the size of the layers just trying to
zoom into this and more accurate the
model set this to a smaller value so
let's let's try setting this to 0.5 and
run it again that's pretty good I'm not
gonna get too hung up on performance I'm
just always curious about this stuff
let's do stop image a scale factor what
to scale the image by oh so I can like
reduce the resolution even more all
right I'm gonna stop tinkering around
with this this is not really super
relevant to what I'm trying to get at
here and I'm gonna put this all back to
what it was
ultimately I just want to run in
processing
okay max pose detection x' let's keep it
a single pose score threshold camera
options with height okay
[Music]
so one of the nice things about working
with runway and its models is a lot of
times models have different parameters
and values and things that you could
tweak and change to try running it in
different ways and this is really these
are sort of known as paper parameters to
a machine learning model and so some of
them I would actually have to stop
running the model and then I can start
to play with it so for example this
architecture one is something that I can
actually make the model smaller it might
be less accurate but will run faster but
but but so for example I'm just gonna
like change this to 0.75 instead I'm
gonna run it again
but some of these some of these
parameters can actually be tweaked in
real time so for example I can change
the width and height of the image which
is actually changing the resolution of
the image from the webcam itself and I
can you know I can make it more
grayscale if I want I can do various
things to actually tweak the image
before it goes in but this is not the
important piece of what I want to do in
this video what I want to do in this
video is we have a moment here where
I've got a model running in runway and
I'm able to play with it tweak it get it
exactly the way I want it to work and I
want to take that next step from having
it run here to be able to see the
results of it in my own piece of
software so let's make that happen
okay this software that I'm going to use
to attempt this in is something called
processing so let's let me write just in
here setup I'm gonna say let me actually
set all this back to default is there
like a set back to default button
feature request but I'm gonna set this
back to 320 by 240 Oh a 320 by 181
that's good let's run it again that'll
be good I just want to like have it
actually let's go
I just want to match the resolution let
me set it back to it yes 3 640 by 360 -
ok
Oh interesting I can mirror if I'm gonna
leave it like that alright so since the
size here in runway of the output is 640
by 360 - what I'm gonna do in my
processing code is set the size of the
canvas to 3 640 by 360 to avoid draw
background 0 alright so now I have a
processing sketch which I am running
right here how do I see the results the
output of the model in my processing
sketch so there are a variety of
different Network protocols that runway
supports and I can find out about them
up here by clicking this network tab
so I'm click this network tab no
interesting sorry I'm like just looking
at things of discovery things as I'm
going so I'm gonna click this network
tab and the one that I want to use for
working with processing is OSC so
there's a variety of different reasons
why you might pick one protocol over
another it really depends on what you're
doing in the case of where I just want
to get a single image an HTTP request
would make the most sense and I'll do
that in another video when I show you
how to work with style gain and run way
but right now I'm going to click on OSC
which works pretty well with processing
and it's telling me a lot of information
here so it's saying hey this is the
server address so this is the most
important
thing that I need from runway it's what
I'm going to tell processing is the
unique IP address which happens to be
the local local IP address of this
computer and the port number from which
it can get the osc messages okay so I'm
going to this is copy this to the
clipboard and I'm gonna click here and
I'm gonna do I'm gonna create a string
called like IP I'm pretty sure I'm
probably going to want the port number
in a separate variable I'm gonna create
a port number like this now I could sit
here and write all the code for this
which is what I usually do in Cody train
videos but this is a differently
different circumstance I really just
want to get the example up and running
and show you how to do this and one of
the nice things about working with
Runway is there a whole bunch of
pre-made examples for you with different
platforms and pieces of software one of
which is processing so let me show you
how you would actually do this in the
real world how I would be doing this
which is the way I'm going to do it
right now so if I go to the runway ml
github and I actually should go back one
level and go here under runway ml you
can see here's the github page for the
runway software and there's a lot of
information this is something you know
there's some like high-level stuff here
about how to port your own machine
learning model to runway itself so if
you've trained your own model or you
find a model that's not supported by
runway how you could add it but that's
not what we're really doing here
what I want to look for is here
processing runway and processing if I
click here this repository has a whole
bunch of examples of using runway with
processing so you can see there's a
street view one attention Gann face lane
landmarks image to text and voila pose
net this is the one that I'm working
with so I'm going to look at this
example I should say that this is an
open source project processing is an
open source project that I'm involved
with that I've talked about in a lot of
other videos so maybe this is a place
where if you find another model in
runway that you've made work and you
want to contribute your processing
example here I would encourage
to do that and Chris one of the founders
and creators of runway and I have been
talking about making a processing
library for runway and it just so
happens that I have made two recent
video tutorials about how to make a
processing java library so I see a
project in the future which is the run a
processing library for a runway so if
you want to get involved with that with
making that you know write in the
comments and let me know that they will
create a github repo for that ok so I'm
gonna go here under pose net and I'm
just gonna click here under pose net pde
so I could just copy paste the whole
thing but I'm gonna kind of go piece by
piece and you can see here already
runway host and runway port so I made a
mistake in my code where I send me this
a string this should really be a number
so I'll do this and there so missing one
here which I'm sure everybody in the
chat I already see it here the dot one
is missing let's import the OSC library
so I'm gonna come over here I'm gonna
import the processing OSC library now am
I getting any errors I'm not getting any
error so I'm lucky for some reason some
dime in my past I needed this library
but you might not be so lucky if you go
here to sketch import library add
library and I search for OSC here it is
oscy p5 and open sound control OSC
implementation you know it looks like I
haven't installed this why don't I have
an error
I'm just curious here for a second pause
for second import library yeah I haven't
installed this why didn't that give me
an error interesting how there's no
error here if I try it lemon well now I
have to satisfy my curiosity here OSC p5
yes ok great
there's my error so nevermind let me let
me back up for a second here for this
eventually edited version of this extra
one in the port I got the port number
wrong oh that one made it in there ok
sorry alright let's just like back all
the way up here for a second I'll redo
this part
so I got it
Oh I'm missing the point one here which
I'm sure the chat is already talking
about and there's an extra one here and
this should be an integer so I have and
this should really be hosts so actually
let me use the same variable names
runway host and runway ports okay so I
think I've gotten this right now now in
order for the example to work I need to
make sure also I have the processing OSC
library installed so if I come over here
I could copy paste these import
statements I can copy paste this OSC
object that I definitely need and we'll
see very quickly that I have an error
which is the class OS CP 5 does not
exist this is because I haven't by are
installed the processing OSC library you
might have already have it installed
because use it with a different project
but I could go here and I can do sketch
import library add library I've already
I've already got it searched to you but
you don't look like this for you you're
gonna want to type OSCE to search for it
this is a library I'm looking for OSC p5
I'm gonna click install it's going to
install and now you'll see that error
message goes away just taking a break
here for a second
all right just okay let's see where are
we oh interesting I've done it this way
with this properties thing that's
interesting let's you won't use that set
s RSP I don't even know what that is do
you not that is I'm just curious let's
look at the OSD p5 documentation and
under examples under we just want to
receive broadcast client I feel like
there's an easy way to just receive send
receive is the example that I usually is
listen for incoming messages so I think
so I wonder if this example could
actually be simplified with just saying
just making a sort of default OS cp5
object rather than this more this OSC
properties object oh I'm really doing
the same thing it's just instead of all
these properties I'm just putting the
port in there so I'm gonna try it and
I'll come back to this if I need to see
if I can simplify this example okay so
let me just look here OSC new OSC yeah
like I actually think you don't even
need the broadcast location because you
don't need to oh you're sending connect
and disconnect messages so you do need
the broadcast location to do that okay
got it got it
so there might be something I'm missing
but we'll see we'll see we'll find out
we'll find out I could all right so next
up
okay where am i processing so the next
thing that I want to do is create an
object to receive OSC messages and I
think I could just say make this a new
OSC pp5 object I need to give it a
reference to this particular sketch
because it's going to need to trigger
events in the sketch when there's data
available and then just give it the port
number if you look at the this is called
runway port so if you look at the runway
example it's actually using this object
called OSC properties which is a bit
more sophisticated it says a remote
address a listening port a Datagram size
and some other stuff I should probably
just copy-paste this as an example but
I'm curious if it will work with just a
sort of more simplified default OS cp5
object where I just say this
and the particular port but I do need
this
I do need a broadcast location because
I'm going to have to send messages to
runway as well saying I'm connected or
I'm disconnected so let's take a minute
to diagram this that did not turn on
this cable there we go
what are the pieces here there's my
processing sketch which I probably sure
I'm gonna write p5 for short I shouldn't
do that because it's too confusing with
p5.js nowadays how are you time we're
getting um so there's two things that
play here
there's the processing software and
there's the runway software they're both
running locally on my computer now it's
possible that in other scenarios there
could also be a cloud GPU involved that
runway sends messages back and forth
between and this is something that I'm
gonna do in the next example where I
work with something called style Gann
I'm going to have runway also talked to
a cloud GPU but that's not happening
here pose net is actually running inside
of basically wrapped into runway itself
locally so posing that's running here in
runway locally processing is sending a
message like connect like hey I want to
hear information that's a one time
message and then runway will just
continuously send data via osc2
processing about what the pose it's
detecting with the pose net model from
the webcam input so if I wanted I
probably could figure out a way to get
the camera input into processing send
the image to runway and then have
runways had the results back but runway
can connect to the camera directly so I
might as well just do that because
that's pretty pretty easy so I should
also put here that the like if I make a
little note here like webcam that webcam
is talking to runway I might also have
it talk to processing if I want to show
the results in processing in a while
okay so this is what's going on in this
particular example so I want to create
this broadcast location which is
a net address object so I need to put
that in here and then the first thing
that I want to do is just send a connect
message so I'm going to copy this in and
paste it here and say connect so what
I'm doing when processing starts up is
it calls the function connect which
creates an OSC message so all oh s see
messages
what's the lingo for this let me look
this up there's like an address and then
a value I think but somebody must
explain this somewhere oscy introduction
to open sound control this will give me
what I'm looking for
[Music]
specification I just wanna know the
lingo data and address I think it's just
address and data so that's that's what
I'm going to that's what I'm going to
use you can find out more here at this
URL okay
every single osc message is comprised of
two parts those two parts are an address
which is usually denoted as a string
kind of like a path and you can see this
is server slash connect that's the
address or you might also think of that
all this is like the message name is the
way I sometimes think about it and then
the data now in this case there actually
is no data because the address itself is
the message so this is a very simplified
thing where this connect message is just
hey I'm connecting so the name of the
message the address is itself all the
only things there's no data but when
runway wants to send stuff back it's
gonna send the message as I forgot what
the name is this is just called message
maybe message keep it anyway what
runways can runways sends data back it's
going to have a message name like data
or key points or poses something like
that and then that's going to have
package with it lots of data like all
the ex-wives of all the positions of
everything data is actually the name of
the message so we'll come back to that
okay all right so let's run this and see
what happens yeah that's that's pretty
good no errors we hold on I think I
generally ignore all these messages that
are in here address already in use
that's weird
I'm not gonna worry right now okay
all right so let's see then alright skip
that part in our nice edit now that I've
connected I should probably add the
disconnect message as well I don't know
that we don't actually need the
disconnect message because it'll
disconnect I'm gonna skip that part okay
now that I've connected I want to listen
for messages and the way that that is
done is with an event called OSC event
so this is much like mousepressed or key
pressed or serial event or capture event
this is a function in processing that
has a very special name called OSC event
and the OSC p5 library knows to call
that function when there's data coming
in so I'm gonna just copy paste this and
I'm going to put it in here and what I'm
going to so let's take a look at this so
there's an OSC event that has passed
through it oh s see message I'll just
change this to message and if the
message has data so this is like its
address remember that it's that address
has data that's the address we're
looking for oh no if it doesn't get out
of here so I won't ignore any other
messages coming in then what I want to
do is get the data itself so the data of
the message actually comes in as a
string but the string is formatted as
JSON which is JavaScript object notation
if you don't know what JSON is I might
refer you to a different video of mine
that explains what JSON is JSON it works
really nicely in JavaScript it's a
little bit awkward to work with it in
processing because processing is Java
doesn't speak JavaScript natively but
we're gonna make it work so the first
thing in the message itself is a big
string of JSON data which then is a JSON
object that can be parsed with
processing parse JSON object function
and then I can just look at it in the
console so let's see if we actually get
the data in
am i sending it I think I might need
that other thing but I was really didn't
think that I need it
okay so this is happening let's see if
any messages are coming in and let's run
that again I have a feeling that there
is something in that thing at the
beginning that I wanted to skip that we
need let's find out so let's go here and
let me change this I think I might know
what the issue is so let me go back to
here and see what this does
oh yeah now it's working so I have a
feeling I know what it is I guess I'll
come back to this in a second and try to
re-explain it so why did that not work
before I don't think it has anything to
do with the Datagram size or this set s
RSP No
oh that's Oh maybe it is bigger than
what it's used to yeah so I don't know
what this I might be knows what this is
but all right so I botched this in my
attempt to want to make this simpler
which I will now correct but
interestingly yeah so I think it was
just the size but also I think it was
kind of freaking out because it was
expecting the processing sketch there's
like something with OS see there's a
client and there's a server and I think
it was expecting the processing sketch
to be the server whereas really run away
as the server the project sketches the
client so the port is different in the
oh did I also get the port run did I
actually put the wrong port in there
that's weird
no but it says ooh weird Oh
is this wrong wait what's going on
oh yeah okay weird so it doesn't need
the data grant the the Datagram sighs so
why is this port 55 seven oh because
have I just confused myself and there's
two different port numbers for sending
and receiving yes okay I am totally
confused or I have to I have to relearn
osc hold on there's there's a couple
things that there's two reasons why it
didn't work
one is I had the wrong port number and
two as I need this to be able to set the
Datagram sighs so that's good we're
gonna be fine this is good we're gonna
I'm gonna explain this and one's gonna
be super happy
okay oh wait oh this is fifty two
hundred wait but I did this this is what
I had right okay because okay it's
because they're on it's all on the same
machine it has to be separate ports okay
I got it now
we're good I'm going to I mean if
everything's gonna make sense in about
ten seconds was they get myself set up
here I'm gonna go back to work I wasn't
getting anything
I'm not getting anything and I'm
wondering why and I actually know why I
had to figure it out so first of all
there's a clue to me here it says could
not create Datagram socket port 5100
because it's already in use and I forgot
there's a weird thing going on here if I
were using OSC to communicate between
two separate computers I could use the
same port number on each separate
computer because it's just one port
number but here I need to be able to
send data to runway at a particular port
as well as receive data into processing
at a different court they cannot be the
same port otherwise it'll be in conflict
so the port that I am broadcasting to is
5 7 100 that's what's listed in runway
but the actual port that I want to
receive messages at is 5 7 200 and maybe
runway knows just to add 100 to it
automatically behind the scenes but this
is the default setup that's in runway so
I need to have each of I need to have a
different port for receiving the data as
the port that I'm sending to and let me
show you what I mean by that in the code
so this is the runway port right there
57 100 that's the port that I want to
broadcast to and it's part of my
broadcast location that's where I'm
broadcasting to but where I want to
receive messages is actually 57 200 so
now if I run this I'm actually receiving
messages but I've got a new error array
index out of bounds exception so this is
a rare case where the data that runway
is sent it sending for all of these
poses is actually quite large and so
what it needs is more space it needs a
bigger packet size and so that's why in
the runway example there was this extra
OSC properties object which allowed
setting a larger Datagram size so in the
listening port is 5 7 200 so just before
using osep 5 by default you don't need
to do this but I'm gonna copy paste this
in I'm gonna put this back here and then
I am going to change this to properties
and I think I now have all the pieces
and what I should see is there we go a
lot of stuff and look at this now I've
got exactly coming into processing a
confidence score for you can see the it
clearly cannot see my left ankle because
my left ankle is not viewable to the
camera so that's why that confidence
score is so low let's scroll up and do
like right here it's got a very high
confidence score and an x and a y for my
right ear so now I'm at the point where
I can actually use this data yeah not
just kind of James and the chat is
pointing out that not doing the
disconnecting step is kind of risky but
I'm gonna not worry about that right now
all right
gotta get to those style gain rainbows
okay that's that's really what we're all
here for know if I oversold it okay but
I'm almost done with this so I think
what I'm gonna do next is I'm gonna I'm
gonna make my own example that just
draws the eyes and then I'll point out
that the the full example does the
entire skeleton alright going back to
the runway example you can see here that
there's a very elaborate loop to parse
through the JSON and look at all the
different key points and get all the
different positions of everything that
it detects I'm gonna try to do something
much simpler right now I'm just going to
get the right eye and left eye so let's
see if we can figure that out so one
thing I want to do is I'm just looking
here okay okay key points is an array
got it so I'd have to loop oh it's it's
interesting an ml five we we change the
pose net API so that you could ask for
something by part name and here it's an
array you have to
loop through it so I will gonna go
through that that's fine all right
that's fine I can actually might as well
loop through all of them if I'm gonna do
that okay okay let's hold on
let me let me start that over again okay
so I might actually end up doing this
anyway get JSON object position okay so
I'm gonna write this code okay let's now
go and so one way so one way we can
approach this is we can make this data
variable a global variable so I'm gonna
take this JSON object and I'm going to
make this a global variable
I'm going to call it data and then in
the draw loop I'm just gonna say as long
as data is not equal to no I forget that
I'm in job I can just do that data will
be null until it's received something
from runway so as soon as it's received
something from runway all I need to do
now is parse this JSON let me put this
in a separate file this will make it
easier give me a second here
let me open up visual studio code
I think I can close all this other stuff
and I can close this pose net JSON and
this alright let's make it easier okay
and then let me
something I've done to make this a
little bit easier is I've just taken
that JSON that printed to the processing
console and I've pasted it pasted it
into a pasted it into a JSON file that I
can look up look at Visual Studio code
just so I have something to reference to
for so I know that I need to get
something called poses so the first
thing that I want is the poses array so
only thing that's really weird and
processing with JSON is you have to
specify whether the data you're looking
at is a JSON object or a JSON array and
this poses data is an array as indicated
by this square bracket so I'm gonna say
JSON array poses equals data get JSON
array string poses and let me just say
then I'm going to then I want to get the
key points array oh because there could
be more than one pose but I'm gonna
assume that's just one pose so actually
what I hold so what what I could do
actually is just say so then that's the
poses so then I'm gonna say the key
points are also an array equal poses dot
get 0 so that would be the first element
of the array now get a new array called
key points get JSON array key points
okay so this the good news is the runway
example has all this in there so if I
get it wrong key points what's the error
I've got
the function Jade get JSON or a string
does not exist poses oh because oh I
have to get get JSON object index 0
sorry I have an error here because I
can't just say get element 0 what is
element 0
it's a JSON object there we go so now I
have the key points which are the JSON
array called key points in the first
JSON object index 0 then what kind of
what do I want to look for whoops so now
I'm in the key points array this is
element 0 if I knew like left left Oh
left this is easy let's do nose left eye
and right eye 0 1 & 2 perfect so I want
to get sorry I want to get JSON object
nose equals key points get JSON object 0
will do will do 3 of these left eye I'm
doing this a little bit different than
the runway example and on point you the
runway example after right eye now
certainly I could use a loop here so
nose I keep going to the wrong place
nose left eye right eye and then I need
to get the position object nose pause
equals nose get JSON object I should
just do the nose I'm just gonna do the
nose just to keep things simpler here
you can you can extrapolate figure out
how to do the left eye and the right eye
nose get JSON object position and then x
equals I'm gonna call it snows position
nose position get X pretty sure this is
right and y equals nose position get Y
all right let's see what have I gotten
wrong here
get float again I'm in Java I've got to
specify the type
so if I've done everything correctly
I've gotten the all the key points of
the first
I've gotten the object with all the data
for the nose then I can get the nose
position out of that object and the X&Y
out of that object whew now I'm gonna
say ellipse X Y 2020 and let's make it a
red nose fill to 5500
let's give this a run and there we go I
am now controlling my nose from runway
into processing with OSC messages
amazing okay so this really concludes
this particular video tutorial certainly
what you might want to do is see the
entire skeleton you know to use pose net
effectively you really want to have the
camera probably around six feet from you
you want to back up and allow it to see
your full form you can also pass the
images and get the pose the skeleton off
from an image there's lots of different
things you can do but and certainly I
would recommend that you check the
example in the runway github repo itself
which has a nice loop to go through all
of the different positions and actually
it also has this little mapping to map
what all the connections are between
them for the actual skeleton skeleton
itself so as a little exercise you know
see if you can expand what I did to have
the right eye and the left eye but even
so you can just go get the runway
example itself so but this is a guiding
principle for how any particular model
that you might find in runway itself
someone in the chat was just asking
about dense pose for example that you
can communicate from runway via OSC to
processing but in a lot of other cases
you might want to use WebSockets or an
HTTP connection to communicate
particularly if you're working in the
browser with JavaScript so what I'm
gonna do in the next video if you want
to watch is run style gand to generate a
rainbow image and then pass that image
into p5 and render it
the browser itself oh and breaking news
from the chat Damion writes SRS P stand
for send and receive on the same port by
default OSC packets are not received and
sent by the same port if you need to
send a receiver on the same port oh so
maybe I could have actually done
something with the port numbers I have
no idea I'm sure people were right about
it in the comments but this code works
and the code in the runway can have
repository works so have fun
use it make something with it please
share it with me and hope you enjoyed
this tutorial about processing runway
and the pose net model running locally
on your computer
goodbye okay this is officially
scheduled to be a two-hour live stream
which means we got about 20 minutes left
this rainbow I mean I probably go a
little long which is fine that's always
happened but I'm gonna do the style game
rainbows so I can use the same port so
in that case I wonder what I wonder
where that other port in is happening in
runway do you know Chris is it like
automatically configuring it to be like
100 up of this server address that's the
default you have in runway like okay so
in future releases of runway you'll be
able to edit the port as well okay in
beta yeah great by the way if this isn't
blowing your mind you haven't tried to
run into this machinery it's really no
my tears asking do I have a flu no I
can't this like nervous tic with the
longer I live stream the more I start to
get the sniffles but I last time I was
sick I had a cold but today I don't all
right let me stop this from running let
me go back to models let me close out of
I'm gonna save this quit processing if
you have questions you can ask them now
I'm going to do one more example and I'm
actually gonna do this
I'm the p5 web editor because I have
done that before shoot I changed the oh
you logged me in perfect I changed the
password at some point okay so usually I
take like a break but I think I'll just
keep going but I'll take a short little
break to read the chat format we'll be
doing the next one any moment now if you
have questions I asked them to check so
I'll answer will answer a few questions
yes so I think that I'm noticing the
chat some people are saying like oh I'm
gonna need to watch this twice to
understand what's going on
this is jam-packed I love it which is a
nice thing to say it's kinda this is a
bit did a little bit different than some
of my live streams in the sense that I'm
you know I have to use runway before I
used it in a presentation at the i/o
conference I've used it in in some
workshops and things that I don't know
that I've taught why don't I talk about
I totally attended workshops at ITP
using runway and I've played around with
a bit by myself but I really think of
this live stream is kind of like a first
exploration of runway and then I might
come back I certainly will come back in
the fall and do some specific tutorials
that are part of my machine learning
course that I'm going to teach with
runway as well as with ml5 GS and
tensorflow Jas and other platforms but I
think that I also might also be
interested think about what would be
most useful in terms of a sort of
sequence tutorial series about runway
and whether that's I'm making it or that
runway itself as a company might release
one but this really is meant to be
somewhat exploratory and so your
feedback about like what makes sense and
what's working for you after you try it
is really useful for me and also
probably also more importantly for the
runway company what's the best place for
people if they want to ask questions or
give feedback on after today the slack
channel is the best place so if you want
to I
if you want to join the slack Channel
I'm gathering that somewhere on the
website here there'd be a link under
resources of the footer okay oh yeah so
here is where you want to go down to the
footer and click here on slack to join
the slack channel and that's a place
where you can ask questions and get
support and you know you know you can
also of course post issues and
contribute on github itself but if
you're beginner a new user to the
software slack it's a great place to do
that okay
[Music]
all right I think it's time now to do
the last tutorial I almost want to
consider this like a coding challenge
video but we'll keep this I think we'll
keep this as a playlist about runway
[Music]
Simon is making a good suggestion which
also could be Simon you could pull
request this I'm sure to the runway
github repo that instead of using
99.999999 there's a variable built-in
variable backs into Duramax ins to get
the maximum space most probably what we
be best would be to find out what's the
exact amount of spaces either so that
you don't waste your space but whatever
works
[Music]
all right it's just let's go let's so I
need to be here and I need the web
editor that's it all I need is that in
the web editor okay and my brain to
understand how to do this okay okay oh
and I had a slide let me pull this up
this might this is gonna take me a
minute let me pull this up oh here's to
oh this is lucky here's the starter code
alright so let's see I want to I my mind
just completely went blank I have no
idea what I'm doing but I was looking
for something oh I know what I was
looking for I was just gonna come up
automatically IO presentation
uh-oh slides oh there we go perfect so
my IO presentation had some slides about
various things including runway and this
is what I want to have so if I do this
I'll just I'll just I'll just reference
this like this that's fine so I just
want to I'll mention this when I come to
it okay okay so I think I have
everything I need right now let's start
with this I'm actually logged in as
something here let's see if there's a
really good one
I like this one I don't know the first
ones probably good enough this one's
kind of nice
alright ready for this coding challenge
okay oh can I just like hold on a sec I
feel a need to do something go away
trends there we go thank you just don't
wanna see that I don't want for the all
of perpetuity anybody comes to watch
this video
about style gain rainbows for them to be
the like obnoxious Twitter trends from
you know arbitrary date of today
whatever's happening in the news that's
horrible
okay here we go this will be the last
last part of today's livestream but
after I finished making this we'll have
some time for some questions afterwards
okay okay hello and welcome to a coding
challenge making style gam rainbows wait
wait hold on no generating oneiza were
generating hello and welcome to a coding
challenge generating style gain rainbows
with a runway ml and passing those two
p5 GS to display them in the browser for
our delight and so this is this is
basically and making a version of
software that I built for the IO
Festival the talk that I gave at the IO
Festival will be out on the Internet at
some point soon and if it is I will
include a link to it in this video's
description where I generally had people
play a game on stage and when they
finished the game on AI rainbow is
generated and tweeted to the IO rainbows
on Twitter account so I'm gonna build a
little mini version of this today so the
and so the primary tool that's gonna do
the generating of the rainbows is
something called runway ml so this is
the first coding challenge that I'm
using with runway and I've talked in a
couple other videos in a livestream more
about what runway is how to download and
install and how to sign up for an
account how to get some free credits and
that sort of thing so I'll refer you to
that video if you want to find
more so but if you have runway
downloaded find a link to download in
the description if you're if you've
signed up for an account and you're on
the Browse models page you will find
yourself here and where you want to go
next is you want to look for style again
now I see style gained right here but
just in case you don't I could type it
in up here under browse models I could
click here this is now giving me
generate photorealistic images of faces
landscape and more there's more
information about the license for style
gaen credits about who created and
authored the original style game paper
that you can find out about but I just
want to use style again so I'm gonna
click Add to workspace and if you don't
have a workspace already you can click
new workspace I have one called coding
train livestream and then Here I am now
I can generate a variety of types of
things with style can and these are
known as checkpoints so there are
checkpoints you can see for cars and
landscapes and portrait wouldn't it be
nice if there was a checkpoint for
rainbows and guess what there is so I'm
gonna click the rainbows checkpoint then
I'm gonna choose the input source which
is just going to be a vector and I'll
talk about what that is in a little bit
as I get to writing some code to do this
and then what I want to do is click run
remotely so this is very important for
me to be clear
this requires cloud credits that you
have to pay for if you sign up through
the link in this video's description
there you can also you'll get free ten
dollars in credits and there's actually
a coupon code coding train which you can
get an additional ten dollars in credits
as well so that's certainly enough to
run this example okay and there's also
in the future there this I think this is
wait timeout for a second this model
only runs remotely yes I'll just say
some models in run way you can run
locally on your computer without using
cloud credits but this one in particular
just runs only runs remotely at the at
the moment so I'm gonna click run
remotely it's running so I'm gonna give
it some time to start up
so generally it's always slow and you're
first starting the model because it's
got it kind of like boot up the instance
also whenever you're alive streaming on
YouTube it takes five times as long as
when you're just sitting at your
computer by yourself but that's the way
the world works all right so now we see
it starting to populate and this is
what's known up here as the latent space
this sort of space of imaginary rainbows
that this generative model is producing
and this is one of the reasons why I
love using runway is that I can actually
just kind of browse around this space
and look for I kind of have this 2d view
of this multi-dimensional world of all
of these rainbows and I can do things
like oh I really like this one and I
could just change the output here to
preview and so I can see it here and I
can download this one and I have now
have my beautiful style game generated
rainbows but what if I want to have
these rainbows in my own software if I
wanted to show them on a web page or if
I want to tweet them from my Twitter
BOTS or any other type of thing that you
might be making whether it's JavaScript
or processing or openFrameworks or some
other piece of software that you want to
connect to runway
the way that you do that is through
talking to run way over the network so
over here in the bottom I'm sorry in the
top right there's a network tab if I
click on this it's showing me a variety
of different options I can communicate
with runway over OSC this is something I
did in a another video tutorial
communicating with processing and runway
over OSC to demonstrate the pose net
model for detecting human skeletal poses
I could also use socket IO for real-time
WebSockets but really what I want to do
is just an HTTP connection I want to
make an HTTP request I want to post some
data to runway and I want to receive the
image back and in fact there's
JavaScript code right here out of the
box that I could copy and paste so I
encourage you to just actually go and
grab this JavaScript code and make your
own example but I'm gonna do this using
the built in p5 function HTTP POST so
I'm gonna write my own code for doing
this referencing everything that's here
under the HTTP option that all is
correct so far I think looking at the
chat alright next get data I'm gonna
post a query I'm gonna post to the query
route and then that will return the
image and this is what I need to post
the array and a truncation number I
forget what truncation is what's it's
like a it's it's kind of like how crazy
it is yeah okay let's see all right
my next step then is to go to the
browser
and I'm going to write this code in the
p5 web editor so first thing that I
might do is let's make a let's make sure
this is running this works okay great
let's make a button Create button
and I'm gonna call that button rainbow
and I'm gonna attach that button to an
event called
whoops let me hit stop here for a second
generate rainbow so when I run it a
button will appear rainbow and then
presumably when I click the button a
function called generate rainbow will be
executed so I need to write that
function and in that function this is
where I want to send my request to
runway itself so I need to make an HTTP
POST so I probably want to look at the
HTTP POST this is a p5 specific function
you could use fetch I have some video
tutorials about how to use the fetch
function to make a post request and you
could do that here but I'm gonna use p5
in this example so I'm gonna go to the
HTTP POST reference page on p5 and this
is gonna show me the stuff that I need
to include in the post request ok I'm
just looking at this so here's the stuff
that I need I need a path where am i
posting this to I need a data type which
is what kind of data am i sending along
with this post request and then any
parameters data that needs to go along
with the post request as well as the
callback and the error callback for
getting information back so this is a
nice I just take this and bring this
into my code I'm just gonna leave this
put this in the comments right here as a
reminder so what are the things the path
is let's go look at runways telling me
this it's actually this the server
address localhost port 8000 so I'm gonna
paste that in here
the data type is JSON that's the kind of
data then the data I'm sending is what
now this is also given to me by runway
and I need to I'm standing a little bit
in front of my code here let me just
move this over so this is the data I
need to send to runway and runways
telling me about that here input
specification what does it expect it
expects field called Z which is an array
of 512 floats what is that and then it
requires some other value called
truncation what is that so if you wanted
to dive deeply into this input
specification you would probably want to
do some more research on the style gain
model itself look at the paper look at
the github repo and kind of understand
more about the neural networks
architecture and its parameters its
hyper parameters that control its
behavior but I whoops
for some reason have lost I'm gonna get
the whiteboard and I think I'm gonna
talk about vectors for a second to
understand this
by the way how do you create the 2d
latent space is it with like teeth knee
or some kind of like like clustering
algorithm like that genetic genetic
algorithms oh that's the greatest thing
I ever heard
okay I'll come back to that all right I
think it's worth though in for a moment
taking a minute in this video tutorial
to talk about what this Z is because
it's a very important concept in machine
learning so there is this machine
learning model called style Gann and it
needs some kind of input in order to
generate some kind of output now the
output that it generates is an image 512
by 512 I mean ultimately what it's
outputting is just a whole lot of
numbers but those numbers can be
interpreted as colors of pixels and
repackaged into the image so that's
happening for you by runway right we're
seeing the output of it right here
packaged as an image but what's the
input I mean ultimately I don't in this
particular example I don't care about
the input I just don't give me a rainbow
give me a rainbow give me a rainbow but
in order for the model to generate a
rainbow it's got to start from somewhere
and in essence I could start with
something random but what that random
thing that I want to start with is is
something called a vector referred to as
Z and what it is is 512 numbers so I
have this list of 512 numbers probably
between 0 and 1 between negative 1 and 1
between 0 & 1
so generally inputs to neural networks
are normalized with some range and in
this in a way this is like a unique
signature for a particular output so if
I want to just get any so-called output
I can just make up a list of random
numbers and I would always get the same
exact rainbow with the same set of
numbers so we can see that happen right
if I fix the set of numbers I'll always
get the same output but what I could do
is tweak these numbers a little bit dial
sum up dial sum
and that's going to change the output if
you've ever and that's what you're
seeing here in this space what you're
seeing here is rainbows with that are
attached to given Z inputs and runway is
being very clever about showing you
similar ones in a two-dimensional flat
space on a computer screen but actually
all of those rainbows that are
generating live in 512 dimensional space
so that's kind of crazy and mind-blowing
and very confusing you know I think I
have a video tutorial where I do
something with four-dimensional space
and I can barely understand that but
this is kind of the weirdness of working
in machine learning it's you could
imagine a three-dimensional space would
just be full of rainbows in 3d like all
over this room there'd be rainbows
everywhere 2d it's just like on a poster
like look at all the rainbows but the
only way to actually literally organize
all of the rainbows generated by style
again would be to have them all sitting
at 512 dimensional space not a thing we
can understand as human beings so that's
why runway cleverly organizing them for
you to look at in two-dimensional space
it's quite useful but you could kind of
walk through that space right I could do
a random walk from vector to vector to
vector in that five dimensional space to
produce an animation of kind of morphing
changing rainbows and that's something
you should really do after you watch
this video and share it with me because
I would love to see that okay but now
that I've explained this let's go back
here now a little more context about
this for any one of these if I really
like this particular rainbow I can
actually get that Z vector let me see if
I can grab it and we're doing it under
export no I just look at it like right
click here on options where is it oh
this this
wait wait wait oh here oh no here right
here right here okay sorry sorry sorry
sir Oh import or export okay so let me
come back to the hugs pretend I know
where it is so it's amazing here is with
this particular rainbow this one that I
liked I can actually go right here and I
can download that 500 those 512 numbers
let me do that and click this I'm gonna
click it correctly there we go looks oh
wait it downloaded sorry so let me click
this and you can see it's saved to this
directory which let me find on my
computer oh where do i click envelope oh
the but i have it has two like i have to
catch it I'm really bad at using a
computer let me try that again make it
very seamless for the edited version of
this video so let me come up Oh was I
saying so one of the wonderful things
about this interface also is that over
here I can export that vector itself so
if I click this here let me try that
again
oh I stopped the model oops
ah what's happened no I'm falling apart
let me restart runway
and I'm gonna go to open workspaces it's
good for us to see all these different
parts of runway I'm back here we run
remotely what's it we're running
something I didn't run the Spade one no
here hi this one got it so if this
happens to you oh this is blocked
because I didn't realize you can't see
it here I should have been doing a
screen capture but that's okay
your if I go down here like I'm Chu Chu
I'll just take myself off for a second
if you this happens to you and you go
you can click here on your profile and
then you can go here and click a stop
all running models which is what I just
did by the way I haven't even spent a
dollar miss entire live stream which
it's really amazing
ok so now let me go back to here I
really wanted to show the vector which
is kind of like not that important
because I'm going to generate random
ones but okay so let me go to here there
we go
there we go um is it not in the
workspace no wait I need to go here
sorry then I need to go here it perfect
I need to say run got it oh we get this
back up and running again I can also
just find that where does it where does
it download stuff your downloads
directory or your documents or sort of
like a runway folder so that is oh here
users coding train documents got it so
let me actually put this on the desktop
and then I'll put it here and their
favorites so that will make things work
really well I disappeared I know I
disappeared I haven't put myself back
yet thank you for reminding that me
though okay so anything is asking can
you even connect to run away from the
browser wouldn't Chrome not allow that
it works there's no reason you can make
a request HTTP requests of the server
running localhost on your computer I
haven't had a problem doing that yet
we'll find out okay so let me go back to
I am in my settings how do I close out
of here if I close this what's going on
here
I think it's trying to stop everything
still okay hold on let's try it one more
time
and open workspace style gam rainbows
run remotely
oh the save folder I didn't actually
change it try that oh yeah I didn't
actually change it to run way open there
we go
and then go back to here and there we go
ooh I like this one oh wow that's wild
oh I like these curvy ones especially
that's pretty cool though
alright so now okay alright and then I'm
gonna go here
all right
this will be a little 510 okay so one of
the nice things about if I'm working in
runway and I find a rainbow that I
really like for example oh I can zoom in
and out that's nuts
this one's kind of crazy-looking if I
like this one
oh look at this strange double rainbow
so let's use this one so I like this one
I can actually click here and export
that vector those 512 numbers as JSON
itself so if I click here and I click
back here I can see this is it this is
that JSON file so I'm just gonna go do
like I'm gonna call this rainbow JSON
let's actually go to the web editor this
is sort of nuts what I'm doing but why
not let's add a file then I'm gonna drop
this file in here and then I'm gonna
look at this and we can see look this is
just that array of numbers and actually
why why even bother making it a separate
JSON file because I'm just gonna say
Const Z equals this array so I actually
just literally copy pasted that array of
numbers looks like by the way it's
between negative 1 and 1 into into my
processing sketch so I'm gonna call this
rainbow Z that now where was I I was
somewhere I was over here in runway
because what I wanted to do was send
that array of 512 floats as the Z
property in the data that I'm sending so
I'm gonna do Z rainbow Z and then I need
truncation so truncation is a hyper
parameter I'd spelled that totally wrong
associated with style Gann if you want
to learn more about truncation you know
that's something you probably just want
to read about in the paper itself but it
kind of changes the craziness way of the
rainbow that you're gonna get and it's a
number I believe that is a number
between
and one and I think the default that's
being used right now my guess is that's
0.5 so it's possible I'm actually gonna
get a different rainbow out if I'm wrong
about that truncation number but now I
have the data then I need a callback and
an error callback so I want to post to a
path I want to post that data type you
know this is sort of silly to have this
separate variable here I can just put a
JSON right in here then I want to post
that data and I want to say got rainbow
or you know got error so I need two
callbacks so now I want to say function
got rainbow data and let's just
console.log the data to see if it comes
back from runway all right we're gonna
run this I'm gonna click the rainbow
button got errors not too fine okay fine
I need to define the got error function
got error error console dot log error
this is so good for some error checking
okay now I'm gonna press this button oh
I got an error what did I get wrong
let's see oh
I'm running the model oh I know what I
got wrong I found what I got wrong so
the server address is localhost port
8000 but I want to make a post request
to the query route so this is actually
what I need as the URL path so I'm gonna
copy this go back to my code we're gonna
hope that this fixes it I'm gonna put
that in here slash query now I'm gonna
hit rainbow look at that so it
console.log something what in the world
if I know you might not believe this but
this is actually a rainbow right here
this is the strangest looking text
version of a rainbow but what's actually
happening there
I'm getting an object only it's got an
image in it but the image is just this
sequence of all these characters so this
has to do with base64 encoding let's go
back to run way to make sure I'm right
about this you can see this is the
output an image and that image is a
base64
image so this is a particular way that
images can be encoded here's one way
that an image could be encoded the first
pixel is a lot of red a little bit of
blue you know I could say start like
describing the image to you pixel by
pixel or I could include an array of
numbers that has the red value the green
value the blue value but it so happens
that what we could do is cleverly take
every color of the image and assign it
to a character in like 64-bit character
encoding by the way I keep seeing these
like alarm bells going off but there's
nothing wrong that was a message from a
while ago I talked about base64 encoding
the whole other video
let me just pull that up here I don't
need to go on and on about base64
encoding yeah
six bit base fixed 64 digits anyway I
don't let me I said 64 bit which is
absolutely wrong so I want to double
back and not say that base64 encoding it
first of all this is something that I've
used it a couple other videos where I've
explained this more thoroughly so I'll
link to that in the video's description
but essentially it's just a way of
encoding all the colors of an image as
ASCII characters so instead of having
numbers for the colors we have unique
characters that that correspond to
certain color values the nice thing
about using base64 is the web speaks
base64 so I can create an image very
easily in javascript in p5 with the
base64 encoding of the image so rather
than console.log that let me try to do
that I'm going to go back to my and that
you could read more also about it on the
base64 Wikipedia page but let me go back
here and I'm just gonna say create image
data image so the property of the image
property that of the data object that's
coming back from run where has the
base64 encoding in it and p5 s create
image function knows how to turn that
into an image element that will appear
on the webpage so let me bring this over
here let me run this again let me hit
rainbow and there it is
look and it's the same one it's the same
one because I gave it this exact vector
but what what might be more interesting
here is why not make a random one each
time so I'm gonna do this when I post
I'm going to create a I'm gonna create a
variable called rainbow Z which is an
empty array I'm gonna loop all the way
up to 512 and I'm gonna say rainbow Z
index I is a random number between
negative 1 and 1 and that's going to be
the rainbow Z so now every time I get a
rainbow press the rainbow button it will
be a different one so now I'm getting
rain random rainbows now here's the
thing they're kind of just by default
making all these Dom elements maybe what
I want to do is actually draw them onto
the
canvas so maybe I'll make the canvas
they happened to be 512 by 512 so I'll
make the canvas 512 by 512 what I'll do
is put this in a variable called rainbow
image I could push them into an array to
save them then I'm gonna say rainbow
image hide so we don't actually see it
but I'll draw it on to the canvas so now
what this is doing is it's creating the
image Dom element from the base64
encoding hiding it from the Dom and then
drawing it onto the canvas so every time
I press this rainbow button what did I
do
let's not hide it oh I think I might
know what the problem is Oh Oh silly
JavaScript and your asynchronous nature
you I think I can't draw the image right
here because it's not actually ready yet
so what I think that I'll do since I
happen to have a draw loop is I'll move
this here and I will make this a global
variable that I will declare at the top
you know this is I could do this in
other ways and then I'll just check as
long as rainbow image exists I will draw
it so now this should give me every time
I click the rainbow button oh and I
still want to hide it Oh every time I
click the rainbow button I get a new
style gand
generated rainbow right here in p5 chess
in the web editor being generated front
runway from the cloud oh we should do a
diagram before I go where's the eraser
here it is this camera still on
alright let's review all the pieces in
this example because there are a lot of
them so I have my own laptop that's
sitting there on the table over there
and there is the web browser running
that's the thing that's running and
there is also the software run way
that's running now Runway has spun up a
local server at localhost 8000
is actually making requests to the p5
web editor server which don't necessary
have to do I could just develop my
JavaScript locally but I'm actually
writing my JavaScript code from the p5
web jazz editor but it is executing and
running that code locally in the browser
so this is kind of not a super important
point but it makes a post request to run
way so when it makes the code makes a
post request to runway runway in turn
runs on the the style gaen model on a
cloud GPU you need to have credits to do
that that is returned back to runway the
resulting rainbow and then sent back to
p5 and rendered in the browser diagram
didn't turn out like I imagined it would
be more interesting but these are the
pieces p5 and runway are both running
locally and they are but but the actual
style game model is running on a runway
server in the cloud that you have access
to through your account now at some
point you might realize well what if I
wanted to create a website where that
would show style gain rainbows I mean
you you can't run run way locally on
your laptop but then a website that's
deployed somewhere else how would you
manage that so if somebody opens up your
p5 sketch it won't work unless they're
running run way themselves on their
local computer but stay tuned I know
that runway is developing features to be
able to deploy your server that's
running the style game model to like a
you know permanent URL in the cloud
somewhere that you could then have your
your JavaScript programming
accessing that other people could run
without having to install runway
themselves so that's the thing that you
could stay tuned and follow in the
future development of runway um the
other thing that's really important for
me to mention here is that this style
gaen model doesn't just exist by
accident so the style gaen architecture
is something that comes from the
original style gaen paper and
pre-trained model and then one of the
creators and founders of runway honest
Ossus hi my name is Jeremiah eNOS I'll
say that again
honest honest are you watching I should
I should do well of Greek names but I
don't um one of the founders and
creators of runway honest us is Jeremiah
neatest actually trained a particular
check point for style Gann with rainbows
and these worked this was trained with
5,000 images tagged with the word
rainbow keyword sorted from relevance
from the Flickr API using this scraping
code to scrape from Flickr from Sam
Levine anti boredom on github so if you
want to find out more about training
your own check point with style Gann I
would refer you to these resources which
I'll include in the video's description
okay
so what are you gonna do with this I
hope that you use this style game
rainbow model for something fun but more
likely hopefully what you're taking away
from this is the fact that you can write
JavaScript code that connects to run way
running a machine learning model that's
actually running in the cloud it could
be writing locally also depending on
which model using from runway if it
supports that and then get send a post
request connect via WebSockets connect
via OSC all sorts um some network
connection to runway get the results
back and use that in your own web
application I would love to see people
figure out interesting ways like how
would you not
how would you generate the rainbow
vectors in such a way that you're kind
of doing a random walk through that
latent space that's five to 512
dimensional space so that's something
you could really think about and play
with and render something out perhaps
you might not even need to use
JavaScript you might be able to do this
even more you know from processing for
example but if you make something with
this please share
go to the kodi trade calm the coding
challenge page associated with this
particular example which you'll find
link to in this video's description and
may we fill the world with more and more
generated rainbows see you soon good bye
[Music]
so okay so I've got one question here
from Nathan in the chat which who asks
why not send the image directly as
binary data instead of as base64 in a
JSON response good I'm reading the
expression on Chris's face and I think
the answer perhaps so you could tell me
if I'm wrong here is good idea I think
base64 is probably chosen as a kind of
standard a web standard a convention
it's very easy to pass around because
you have you're able to take so much
information impact packet into one
character but I could imagine there are
other scenarios where you might want
that binary data so that could
potentially be a future feature of
Runway and I would encourage you to join
the flack channel and suggest it there
what other questions do people have so I
think we can take maybe five or ten
minutes here to to answer any questions
you can ask them the chat Chris can type
them directly or I don't want to just at
least come and wave hello people could
see your face or he could come up an
answer I think my mic will pick him up
but we could take a few questions before
I go yes Chris can you hear Chris if he
talks yeah so we'll see that in the chat
so let's see so if you are in this for
for coding train members or patrons
there's a slack channel you can ask
stuff in there which I'm looking at and
also keeping an eye on the YouTube chat
now this is the awkward thing where we
just stand around because there are 30
seconds behind I'm thinking of
deployment won't that open up to other
people spending all your credit yes no
we're working on ways of deploying your
models to production so you won't spin
your credit so will work in a different
way and then way you have like a remote
URL that will always be able
for you until you kind of think stop it
and then we'll be different from running
okay yeah but in a way it's no different
than if you have to like spin up a web
server for your web application and you
have to pay hosting fees for that this
is just like sort of new kind of hosting
GPU fee that you have to pay to run your
web application but ideally there will
be in the you know as many open source
or ways to run your own so like reduce
that cost or make it sustainable for
independent Cree and as an artists I
think that's something that runway is
working towards which is different than
what you know large corporations can do
with their giant server farms would you
use Perlin noise for that Oh interesting
I don't know what you're asking exactly
but generating the inputs with Perlin
noise could be kind of interesting it's
not that you wouldn't what what you
might discover from that I would think
of it less as like you're using the
Perlin noise space to fill a vector but
you might use Perlin noise to move from
one vector to another so that you're
just like subtly changing all the
individual values a little bit so your
next rainbow is pretty similar to the
previous one but that would be like a
way of having a random another way of
thinking about a random walk somebody
asked for like recommendations of books
to learn machine learning you have any
of it we're really helpful for you I do
know why you're thinking I do know that
there's I don't know if it's out already
or if you could just uh but there's one
that's the deep learning with tensorflow
J s book that's like I think you can get
the like book the meat which is like the
yeah deep learning with JavaScript so I
think this book is coming now oh and the
Python version is excellent so there's a
Python version by francois la the
creator of chaos which i have definitely
used that's been excellent okay whoa
spin for a chance to win all right let's
do it everybody manning publications not
a sponsor
boy they're so good with our advertising
techniques
I got 50% off not entering my email
sorry Manny okay any other questions so
let's see what other things do I want to
say before we go so number one if you
have more questions about runway or you
want to teach a workshop with runway or
you want to get involved with runway
apply for a job contribute to some of
the open source components of runway
then I would suggest I'm going to show
you again
signing up here under slack what do I
get if I click on community okay okay
yeah if you happen to be in the New York
City area runway hosts periodically
meetups at their offices that I've been
to at least 50% of them and it was
pretty awesome so I highly recommend
keeping an eye on that I know that the
School of Law I think is using oh yeah
is it this bots 1 so this is an
independent artist run school in I
believe based in Berlin that is
currently running a workshop I think
inning she is teaching it who you might
have remember from previous episodes of
the coding train it's teaching that and
then it looks like there's another
upcoming one in Amsterdam
whoo July do I have time to make it to
Amsterdam by next week I would be
awesome
I can't this month is so crazy we gotta
gotta move ITP to a new building but and
then this was the runway Meetup
that was in Brooklyn and there's some
other ones listed here so yeah let's see
can yer it's a good good fella's book is
free Thank You K week Vaughn
do you think your random walker tutorial
would be useful for 512 dimensions yes
so ultimately what I'm suggesting and I
haven't tried this and I'm certainly not
an expert on generating you like
artistic renderings of you know machine
learning models generating images in
Layton States but if you want to
actually look at some artists who do
this really beautifully memo Aachen is
one that comes to my mind I don't know
if just on his homepage if stuff will
come up instant instantly but he I'm a I
machine learning maybe is where I would
want to click here but this is
definitely an artist who's done a lot of
work in that area I know that yg is an
ITP current student in my recent video
about the ITP show had a project with AI
generated cats like a little movie
generated cats and Layton space but all
that is to say that the random walk
which takes an XY point and adds random
numbers to the X and random numbers to
the Y to get the next point all you need
to do is apply that same thinking to 512
X Y Z u V but we're gonna run out of
letters so just think of instead of x
and a y as an array of two numbers but
instead of an array of two numbers an
array of 512 numbers that's the right
idea right and probably you could do
some interesting things about wandering
that space and coming back to where you
started so that you could create a sort
of perfect loop so there could be ways
of thinking about doing that and gene
Kogan is on also I I sort of like didn't
reference gene Cogan because that's kind
of like but of course but you might not
be from ml for a ml for a website has a
lot of tutorials and resources I'm not
sure if any of these are currently using
runways some of them are but this is a
really amazing resource for getting
started with machine learning for
artists in theory if I can get my act
together I'm teaching a course at NYU
it's an undergraduate course called
introduction to machine learning for the
Arts and so I'll be going through a lot
of different projects with primarily
with the main things I'm going to use
are tensorflow J s ml v GS and runway so
hopefully I'll be making more
possibly with the YouTube learning
playlist which I need to talk about
before I go so I'm just looking for how
to access Mouse code for Mac sorry old
fire craft I'm not sure exactly what
you're asking I'm also unfortunately
trying to limit the questions now to
runway related stuff but that's a really
great question you could ask at
discourse processing org and you're
always welcome to tweet me a link to
your post or post a link to your post in
a comment that I can take a look at it
as well update on the promo code what's
that referring to Neath and I don't know
what that's referring to the promo code
is just coding train if you're asking
for the promo code for runway okay
[Music]
okay so uh so thank you Chris so much
for being here there are many other
people who are involved with runway and
community and developing a design so
check out the runway website to find out
all those people and yeah the chat will
run for a little while let the music
play and let the livestream go for a
little bit so Chris will still be in the
chat if you have questions um the last
thing that I want to mention before I go
which is super exciting I wasn't able to
say this before because it wasn't
publicly announced but let's see if this
happened so I'm gonna go to the home
page for the coding train and I'm just
gonna like arbitrarily click on look at
this this is new it's actually set this
here full learning playlist so there's a
new feature on YouTube called learning
playlist it's in beta I was in the
learning playlist beta to create some of
my to put some of my existing content
into the learning playlist but more
notably probably worse where's the pop
pop I don't want popular uploads Owen
uploads this particular this particular
set of videos that were from the working
with data and api's and JavaScript
playlists are part of learning plan so I
just want to show you some of these
features I'm not logged in because I
YouTube premium so I don't have to watch
these ads that I've always tried to be
logged in nope - lean
go on you can do it no it's showing the
path did you all see that that's the
words I click the eye and I just showed
my vest I do this constantly I really
have to chip that's bad
hold on how do I change my Google
password just be nice people please
don't ruin the internet for all of us
security oh I do it let you step
verification that's the one account that
I don't actually have it in good success
I changed my password before any of you
can you know what's so great about the
30 second lag no wonder you weren't able
to hack it you didn't see it oh geez
what a nightmare
oh now my email is up oh no button where
the screams not on okay great well I'm
glad I have this button okay here we are
back on YouTube and you could see now
that there's this link here learning
playlist and also browse all learning so
one thing I'll point out if you go to
that this is YouTube's learning channel
with a lot of other learning playlist
I definitely would recommend this one
from Jibril's about everything you need
to start programming it's pretty awesome
somewhere on here maybe I'm featured if
I'm lucky introduce oh there I am
code program with p5 yes if you like
this learning playlist or you just want
to like be nice to me you could tweet at
like they're really like you could tweet
with the hashtag I don't know YouTube
learning I think are @yt creators say
how much you like the coding trains new
learning playlists or something that you
like beg for this kind of stuff on the
internet it's very very unbecoming of me
but anyway so I I'm excited about this
feature if I click on it you can see
there's a new interface so the course
trailer pops up here you can see the
different sections are divided showing
you how many videos there are on
actually I you know you can save it so
it'll like keep track of you in it what
are your progress in the in the course
or the playlist and so I recommend you
taking a look at those oh it's
interesting it's my intro to code one
that's featured there and actually
crazily where's that YouTube blog so if
I go to Twitter and I go to it's kind of
blew my mind even though nobody noticed
it somewhere here learning playlists boy
they tweet too much there's this whole
like VidCon conference that I completely
missed here it is so here if you go to
this tweet and you click on here umm
this official blog from YouTube from
Thursday July 11th if you read this
through then you read about all this
other nonsense stuff and look at this
there's this chemistry course from Hank
green you can see the screenshot of the
learning playlist you can click right
here this is this is the experience I
had read August love cuz it's like oh
cool they're announcing the learning
playlist oh and they're featuring
chemistry oh and a working in Java
playlist oh cool they're featuring a
programming one I wonder who made a Java
playlist because I certainly did make a
Java playlist let me click on this link
to find out what YouTube channel has a
java learning to code in Java playlist
so then I clicked on that link and it
took me here and I was like wait no oh I
must have missed clicked because this is
my playlist and it's JavaScript let me
go back to that and let me click again
and then I ended up here and I realized
oh they actually linked to my playlist
on the official YouTube blog but they
kind of got the topic wrong but that's
fine I'm not one to complain thank you
for linking to be may be very happy
alright nobody has my email nobody's
reading my email nobody has my password
right we're good can you let me know in
the chat if you've hacked my account I'm
just gonna check just gonna check really
quickly it's funny I have to factor on
for everything and I think I'm gonna
turn that on right except for my Daniel
at coding training account because I
don't actually use that one but
apparently now I do use it so I'm gonna
go to security
I'm gonna turn on you don't mind you ask
questions about runway in the chat but
it's gonna be here doing my two-factor
authentication I'm turning that on now
too so this doesn't happen in the future
no can you hear that Oh your password
was changed less than an hour ago I
guess I'll have to do it later okay so
if your password was changed less than
an hour ago it doesn't let you add
two-factor authentication
that's fine you should really use a
password manager go back and find an old
video on the coding train that had a
sponsor that was a password manager
because I've never seen this before
your password has changed less than an
hour ago and find that affiliate link
it's not an affiliate link it's just a
link whatever okay that's right okay
I'll be able to cut that out very
quickly later all right
so unfortunately this live stream is now
ending and I'm gonna have to make it fit
very briefly it's like that YouTube let
lets you like cut out a frame and
goodbye
thank you for watching I will see you
later
uh so I don't know when I'm coming back
stay tuned to my Twitter stay tuned to
other stuff check out the community tab
if you're if you're a member or patreon
I've got some stuff that I'm thinking
about changing the rewards program so
stay tuned I hope to see you in a future
livestream look for some of the new
videos coming out soon and I will I
might be back but I will definitely oh
no I definitely will be back but I might
not be back until late August or early
September but if you see a notification
pop up that there's a livestream that
means I'm doing a special surprise
livestream so see you soon and good bye
everybody
[Music]