[TRAIN HORN]
Hello.
And welcome to a coding challenge, Generating StyleGAN
Rainbows With Runway ML and passing those to P5 JS
to display them in the browser for our delight.
And so this is basically making a version of software
that I built for the I/O Festival.
The talk that I gave at the I/O Festival
will be out on the internet at some point soon.
And if it is, I will include a link
to it in this video's description
where I had people play a game on stage.
And when they finished the game, an AI rainbow
was generated and tweeted to the I/O Rainbow's on Twitter
account.
So I'm going to build a little mini version of this today.
And so the primary tool that's going
to do the generating of the rainbows
is something called Runway ML.
So this is the first coding challenge
that I'm using with Runway.
And I've talked in a couple other videos
in a livestream more about what Runway
is, how to download and install it,
how to sign up for an account, how
to get some free credits, and that sort of thing.
So I'll refer you to that video if you want to find out more.
But if you have Runway downloaded--
find a link to download it in the description.
If you've signed up for an account,
and you're on the Browse Models page,
you will find yourself here.
And where you want to go next is you want to look for StyleGAN.
Now, I see StyleGAN right here.
But just in case you don't, I could type it in up here
under Browse Models.
I could click here.
This is now giving me, "Generate photorealistic images
of faces, landscape, and more."
There's more information about the license for StyleGAN,
credits about who created and authored the original StyleGAN
paper that you can find out about.
But I just want to use StyleGAN.
So I'm going to click Add to Workspace.
And if you don't have a workspace already,
you can click New Workspace.
I have one called Coding Train Livestream.
And then here I am.
Now, I can generate a variety of types of things with StyleGAN.
And these are known as checkpoints.
So there are checkpoints, you can see,
for cars, and landscapes, and portrait.
Wouldn't it be nice if there was a checkpoint for rainbows?
And guess what.
There is.
So I'm going to click the rainbows checkpoint.
Then I'm going to choose the input source, which
is just going to be a vector.
And I'll talk about what that is in a little bit
as I get to writing some code to do this.
And then what I want to do is click Run Remotely.
So this is very important for me to be clear.
This requires cloud credits that you have to pay for.
If you sign up through the link in this video's description,
you'll get free $10 in credits.
And there's actually a coupon code, CODINGTRAIN,
which you can get an additional $10 in credits as well.
So that's certainly enough to run this example.
Some models in Runway you can run locally on your computer
without using cloud credits.
But this one in particular only runs remotely at the moment.
So I'm going to click Run Remotely.
It's running.
So I'm going to give it some time to start up.
So now we see it's starting to populate.
And this is what's known up here as the latent
space, this sort of space of imaginary rainbows
that this generative model is producing.
And this is one of the reasons why
I love using Runway is that I can actually just kind
of browse around this space, and kind of
have this 2D view of this multi-dimensional world of all
of these rainbows.
And I can do things like, oh, I really like this one.
And I could just change the output here to preview.
And so I can see it here.
And I could download this one.
And I now have my beautiful StyleGAN generated rainbows.
But what if I want to have these rainbows in my own software?
If I wanted to show them on a web page?
Or if I want to tweet them from my Twitter bot?
Or any other types of thing that you
might be making, whether it's JavaScript, processing,
open frameworks, or some other piece of software
that you want to connect to Runway.
The way that you do that is through talking to Runway
over the network.
So over here in the bottom--
I'm sorry.
In the top right, there is a Network tab.
If I click on this, it's showing me
a variety of different options.
I can communicate with Runway over OSC.
This is something I did in another video tutorial,
communicating with processing and Runway over OSC
to demonstrate the PoseNet model for detecting
human skeletal poses.
I could also use Socket I/O for real time web sockets.
But really what I want to do is just an HTTP connection.
I want to make an HTTP request.
I want to post some data to Runway.
And I want to receive the image back.
And, in fact, there's JavaScript code right here out of the box
that I could copy and paste.
So I encourage you to just actually
go and grab this JavaScript code,
and make your own example.
But I am going to do this using the built in P5
function, HTTP Post.
So I'm going to write my own code
for doing this, referencing everything that's
here under the HTTP option.
My next step then is to go to the browser.
And I am going to write this code in the P5 web editor.
So first thing that I might do is
let's make sure this is running, if this works.
OK.
Great.
Let's make a button.
Create Button.
And I'm going to call that button rainbow.
And I'm going to attach that button to an event called--
whoops.
Let me hit stop here for a second--
Generate Rainbow.
So when I run it, a button will appear.
Rainbow.
And then presumably when I click the button,
a function called Generate Rainbow will be executed.
So I need to write that function.
And in that function, this is where
I want to send my request to Runway itself.
So I need to make an HTTP post.
So I probably want to look at the HTTP post.
This is a P5 specific function.
You could use fetch.
I have some video tutorials about how
to use the fetch function to make a post request.
And you could do that here.
But I'm going to use P5 in this example.
So I'm going to go to the HTTP post reference page on P5.
And this is going to show me the stuff that I need
to include in the post request.
So here's the stuff that I need.
I need a path.
Where am I posting this to?
I need a data type, which is what kind of data
am I sending along with this post request?
And then any parameters, data that
needs to go along with the post request, as well
as the callback and the error callback for getting
information back.
So if I could just take this and bring this into my code,
I'm just going to put this in the comments
right here as a reminder.
So what are the things?
The path is--
Runway's telling me this.
It's actually this.
The server address.
Local host port 8000.
So I'm going to paste that in here.
The data type is JSON.
That's the kind of data.
The data I'm sending is what?
So this is the data I need to send to Runway.
And Runway's telling me about that here.
Input specification.
What does it expect?
It expects a field called Z, which
is an array of 512 floats.
What is that?
And then it requires some other value called truncation.
What is that?
So if you wanted to dive deeply into this input specification,
you would probably want to do some more research
on the StyleGAN model itself.
Look at the paper.
Look at the GitHub repo.
And kind of understand more about the neural network's
architecture, and its parameters,
its hyper parameters that control its behavior.
I think it's worth, though, for a moment taking
a minute in this video tutorial to talk about what
this z is, cause it's a very important concept in machine
learning.
So there is this machine learning model called StyleGAN.
And it needs some kind of input in order
to generate some kind of output.
Now, the output that it generates is an image, 512x512.
I mean, ultimately what it's outputting
is just a whole lot of numbers.
But those numbers can be interpreted as colors of pixels
and repackaged in the image.
So that's happening for you by Runway.
Right?
We're seeing the output of it right here,
packaged as an image.
But what's the input?
I mean, ultimately, in this particular example,
I don't care about the input.
I just want, give me a rainbow.
Give me a rainbow.
But in order for the model to generate a rainbow,
it's got to start from somewhere.
And in essence, I could start with something random.
But what that random thing that I want to start with
is is something called a vector, referred to as z.
And what it is is 512 numbers.
So I have this list of 512 numbers,
probably between 0 and 1.
So generally inputs to neural networks
are normalized with some range.
And in a way, this is like a unique signature
for a particular output.
So if I want to just get any so-called output,
I can just make up a list of random numbers.
And I would always get the same exact rainbow
with the same set of numbers.
So we could see that happen.
Right?
If I fix that set of numbers, I'll
always get the same output.
But what I could do is tweak these numbers a little bit,
dial some up, dial some down.
And that's going to change the output.
And that's what you're seeing here in this space.
What you're seeing here is rainbows that
are attached to given z inputs.
And Runway is being very clever about showing you
similar ones in a two dimensional flat space
on a computer screen.
But actually, all of those rainbows that are generated
live in 512 dimensional space.
So that's kind of crazy, and mind blowing,
and very confusing.
I think I have a video tutorial where I do something
with four dimensional space, and I can barely understand that.
But this is kind of the weirdness
of working in machine learning is
you could imagine a three dimensional space would just
be full of rainbows in 3D.
Like all over this room, there'd be rainbows everywhere.
2D.
It's just on a poster, like look at all the rainbows.
But the only way to actually literally organize
all of the rainbows generated by StyleGAN
would be to have them all sitting
in 512 dimensional space.
Not a thing we can understand as human beings.
So that's why Runway cleverly organizing them
for you to look at in two dimensional space
is quite useful.
But you could kind of walk through that space.
Right?
I could do a random walk from vector, to vector,
to vector in that five dimensional space to produce
an animation of kind of morphing, changing rainbows.
And that's something you should really
do after you watch this video, and share it with me.
Cause I would love to see that.
So one of the nice things about if I'm working in Runway
and I find a rainbow that I really like, for example--
oops.
I can zoom in and out.
This is nuts.
This one's kind of crazy looking.
If I like this one, oh, look at this strange double rainbow.
So let's use this one.
So if I like this one, I can actually
click here and export that vector,
and those 512 numbers as JSON itself.
So if I click here, and I click back here,
I could see this is it.
This is that JSON file.
So I'm just going to go call this rainbow.JSON.
Let's actually go to the web editor.
This is sort of nuts what I'm doing.
But why not?
Let's add a file.
Then I'm going to drop this file in here.
And then I'm going to look at this.
And we can see, look, this is just that array of numbers.
And actually, why even bother making it a separate JSON file?
Because I'm just going to say const z equals this array.
So I actually just literally copy
pasted that array of numbers.
It looks like, by the way, it's between negative 1 and 1.
Into my processing sketch.
So I'm going to call this rainbow z.
Now, where was I?
I was somewhere.
I was over here in runway, because what I wanted to do
was send that array of 512 floats
as the z property in the data that I'm sending.
So I'm going to do z, rainbow z.
And then I need truncation.
So truncation is a hyper parameter--
I spelled that totally wrong--
associated with StyleGAN.
If you want to learn more about truncation,
that's something you probably just want
to read about in the paper itself.
But it kind of changes the craziness factor, in a way,
of the rainbow that you're going to get.
And it's a number.
I believe that is a number between 0 and 1.
And I think the default that's being used right now,
my guess is that it's 0.5.
So it's possible I'm actually going
to get a different rainbow out if I'm
wrong about that truncation number.
But now I have the data.
Then I need a callback and an error callback.
So I want to post to a path.
I want to post that data type.
This is sort of silly to have this separate variable here.
I can just put JSON right in here.
Then I want to post that data.
And I want to say, got rainbow, or got error.
So I need two callbacks.
So now I want to say, function, got rainbow.
Data.
And let's just console log the data
to see if it comes back from Runway.
All right.
We're going to run this.
I'm going to click the Rainbow button.
Got error is not defined.
OK.
Fine.
I need to define the got error function.
Got error.
Error.
Console.log.
Error.
This is good for some error checking.
OK.
Now I'm going to press this button.
Ooh.
I got an error.
[BUZZER]
[DING]
I found what I got wrong.
So the server address is local host port 8000.
But I want to make a post request to the query route.
So this is actually what I need as the URL path.
So I'm going to copy this.
Go back to my code.
We're going to hope that this fixes it.
I'm going to put that in here.
Slash query.
Now I'm going to hit rainbow.
Ah!
Look at that.
So it console logged something.
What in the world?
I know you might not believe this,
but this is actually a rainbow right here.
This is the strangest looking text version of a rainbow.
But what's actually happening there?
I'm getting an object.
Oh, and it's got an image in it.
But the image is just this sequence
of all these characters.
So this has to do with base 64 encoding.
Let's go back to Runway to make sure I'm right about this.
You can see this is the output.
And image.
And that image is a Base64 image.
Base64 encoding, first of all, this
is something that I've used in a couple other videos
where I've explained this more thoroughly.
So I'll link to that in the video's description.
But essentially, it's just a way of encoding
all the colors of an image as ASCII characters.
So instead of having numbers for the colors,
we have unique characters that correspond to certain color
values.
The nice thing about using Base64
is the web speaks Base64.
So I can create an image very easily in JavaScript,
in P5 with the Base64 encoding of the image.
So rather than console log that, let me try to do that.
And you could read more also about it
on the Base64 Wikipedia page.
But let me go back here.
And I'm just going to say create image data image.
So image property of the data object
that's coming back from Runway has the Base64 encoding in it.
And P5's create image function knows
how to turn that into an image element that
will appear on the web page.
So let me bring this over here.
Let me run this again.
Let me hit rainbow.
And there it is.
Look.
And it's the same one!
It's the same one, because I gave it this exact vector.
But what might be more interesting here
is why not make a random one each time?
So I'm going to do this.
When I post I'm going to create a variable
called rainbow Z, which is an empty array.
I'm going to loop all the way up to 512.
And I'm going to say rainbow Z index
I is a random number between negative 1 and 1.
And that's going to be the rainbow Z.
So now every time I get a rainbow,
press the rainbow button, it will be a different one.
So now I'm getting random rainbows.
Now, here's the thing.
They're kind of just, by default,
making all these dumb elements.
Maybe what I want to do is actually draw them
onto the canvas.
So maybe I'll make the canvas.
They happen to be 512x512.
So I'll make the canvas 512x512.
What I'll do is put this in a variable called rainbow image.
I could push them into an array to save them.
Then I'm going to say rainbow image hide.
So we don't actually see it.
But I'll draw it onto the canvas.
So now what this is doing is it's creating the image dom
element from the Base64 encoding,
hiding it from the dom, and then drawing it onto the canvas.
So every time I press this rainbow button--
[STATIC]
Oh, silly JavaScript and your asynchronous nature you.
I think I can't draw the image right here, because it's not
actually ready yet.
So what I think that I'll do, since I happen to have a draw
loop, is I'll move this here.
And I will make this a global variable
that I will declare at the top.
I could do this in other ways.
And then I'll just check.
As long as rainbow image exists, I will draw it.
So now, this should give me every time
I click the rainbow button--
oh, and I still want to hide it.
Whoa.
Every time I click the rainbow button I get a new StyleGAN
generated rainbow right here in P5 JS
in the web editor being generated
from Runway from the cloud.
Oh, we should do a diagram.
Let's review all of the pieces in this example
because there are a lot of them.
So I have my own laptop that's sitting there
on the table over there.
And there is the web browser running.
That's a thing that's running.
And there is also the software Runway that's running.
Now, Runway has spun up a local server at local host 8000.
The browser is actually making requests
to the P5 web editor's server, which you don't necessarily
have to do.
I could just develop my JavaScript locally.
But I'm actually writing my JavaScript code
from the P5 Web JS editor.
But it is executing and writing that code locally
in the browser.
So this is kind of not a super important point.
But it makes a post request to Runway.
So when the code makes a post request to Runway,
Runway, in turn, runs on the StyleGAN model on a cloud GPU.
You need to have credits to do that.
That is returned back to Runway, the resulting rainbow,
and then sent back to P5 and rendered in the browser.
This diagram didn't turn out like I imagined it.
So I thought it would be more interesting.
But these are the pieces.
P5 and runway are both running locally.
But the actual StyleGAN model is running
on a Runway server in the cloud that you have access
to through your account.
Now, at some point you might realize, well,
what if I wanted to create a website where that
would show StyleGAN rainbows.
I mean, you can't run Runway locally on your laptop.
But then a website that's deployed somewhere else,
how would you manage that.
So if somebody opens up your P5 sketch,
it won't work unless they're running Runway themselves
on their local computer.
But stay tuned.
I know that Runway is developing features
to be able to deploy your server that's running the StyleGAN
model too, like a permanent URL in the cloud
somewhere, that you could then have your JavaScript
programming accessing that other people could run without having
to install Runway themselves.
So that's something that you could
stay tuned and follow the future development of Runway.
The other thing that's really important for me
to mention here is that this StyleGAN model doesn't just
exist by accident.
So the StyleGAN architecture is something
that comes from the original StyleGAN
paper and pre-trained model.
One of the founders and creators of Runway,
Anastasis Germanidis, actually trained a particular checkpoint
for StyleGAN with rainbows.
And this was trained with 5,000 images tagged with the word
rainbow keyword, sorted for relevance from the Flickr API
using this scraping code to scrape from Flickr,
from Sam Levine, AntiBoredom on GitHub.
So if you want to find out more about training
your own checkpoint with StyleGAN
I would refer you to these resources, which I'll include
in the video's description.
OK.
So what are you going to do with this?
I hope that you use this StyleGAN rainbow
model for something fun.
But more likely, hopefully what you're taking away from this
is the fact that you can write JavaScript code that
connects to Runway running a machine learning model that's
actually running in the cloud.
It could be running locally also,
depending on which model you're using from Runway if it
supports that.
And then send a post request.
Connect via WebSockets.
Connect via OSC.
Some network connection to Runway.
Get the results back.
And use that in your own web application.
I would love to see people figure out interesting ways.
Like how would you generate the rainbow vectors in such a way
that you're kind of doing a random walk through that latent
space, that 512 dimensional space.
So that's something you could really
think about and play with, and render something out perhaps.
You might not even need to use JavaScript.
You might be able to do this even more, from processing,
for example.
But if you make something with this, please share it with me.
Go to thecodingtrain.com, the coding challenge
page associated with this particular example, which
you'll find linked to in this video's description.
And may we fill the world with more and more
generated rainbows.
See you soon.
Goodbye.
[TRAIN HORN]
[MUSIC PLAYING]