(tooting train whistle)
Hello, welcome to another holiday coding challenge!
Again with the snowflakes!
This is my second snowflake challenge,
but in this time instead of creating an algorithm
to draw a snowflake, I'm going to ask a neural network
to draw a snowflake for me.
Now, there's a lot of pieces to all of the work
that's gone in to making the fact
that I'm going to be able to do this
in well what probably will be about four and a half hours,
but really could take just about 15 minutes possible.
The first thing I want to reference is the Magenta Project.
So the Magenta Project is a make music and art
using machine learning research project,
it's an open source research project from Google,
and it has a ton of examples,
projects with TensorFlow in both Python and JavaScript,
and you can look through at all these featured projects.
The project I want to mention is, I'll just click over here,
is this project called SketchRNN,
or draw together with a neural network.
Now, if you've watched any of my previous videos
about the Quick, Draw dataset,
the Quick, Draw dataset is this huge collection
of millions upon millions of doodles
that users from around the world made,
playing the Google Quick, Draw game.
So what the researchers from Google Brain did,
you can see them credited here David Ha,
Jonas Jongejan, and Ian Johnson
wrote this paper about how a neural network
could be trained on all of those drawings,
so it could learn okay when somebody's drawing a cat
they kind of go like this, and then they go like this,
and it could start to imagine new drawings of cats.
And it's called SketchRNN because the kind of neural network
that it's using is something called
a Recurrent Neural Network.
And you might have also seen this,
Nabil Hassein was here and did a guest video
about generating texting with a Recurrent Neural Network.
A Recurrent Neural Network is good
at learning about sequential information,
like hello world is sequential information.
H-E-L-L-O space W-O-R-L-D.
Music is sequential information (singing a scale)
that's a sequence of notes and rhythms.
A drawing is a sequence of vectors.
So you can read this paper all about how they used
the Google Quick, Draw dataset
to train a model to create new drawings.
You could also look at their demo,
and this is the code by the way for their demo.
There's a lot involved here in all of these pieces,
and here's the demo right here.
So, I'm going to pick cat,
and what I can do is I can start drawing,
oops, I can start drawing a cat,
I'm just going to draw a circle,
and then it will pick up for me
and finish the rest of the cat.
So at some point I will do a part two of this video
where I actually create an interactive version like that,
but one of the things that is in now the ml5 project,
ml5 is a friendly machine learning for the web library
that's built on top of TensorFlow.js.
It works with p5 which is the open source JavaScript library
that I use in a lot of my video tutorials,
is it has a SketchRNN module for you.
So on the one hand you might want to go
and look all the way through all this code
and read the research paper to learn how it works,
but if you want to quickly get up and running
with playing with a SketchRNN model,
what we've tried to put in ml5 will help you.
Now, (laughs) you might notice that if I go
click on reference, at the time of this recording
it doesn't say SketchRNN here.
So this is actually a new feature.
There's a couple issues in the GitHub repository
for ml5 about things that don't work perfectly,
or need to be added, or fixed, we welcome contributions.
I will come back and maybe do some more videos with it
once it's gotten further along and there is documentation
but for the holiday season let's see
if we can get it to draw a snowflake for us.
So, I'm not going to be able to use the website
because the documentation isn't there,
but I've got some documentation
over here on my invisible computer (laughs)
that I can look at if I need to.
So what do I have?
I have a blank p5 sketch with just setup and draw,
and also at index.html I should call this
Snowflake SketchRNN.
I have also in addition to the p5 libraries being imported,
I have the ml5 library and again
at the time of this recording it's version 0.1.3.
Hopefully there'll be future updates to improve SketchRNN
and there'll be a higher numbered version.
So the first thing that I want to do
is I want to make a variable and I'm going to call it model.
So model, I could call it SketchRNN,
maybe I should call it SketchRNN,
I'll just call it model, model is the thing
that's going to hold the SketchRNN model.
It is all of the information,
the brain of that neural network
that we can ask to generate new stuff.
So we can say, hey give me a new point
along the path that you're imagining to draw.
So I'm going to say model equals ml5
SketchRNN,
and then I'm going to just put a callback in here, model ready,
and this is the important thing, what goes here.
So there are a bunch of available models with SketchRNN
and we can see that list here, whoops, here.
For example there is cat, let's start with cat maybe
but we can see all the other ones like
the Mona Lisa apparently is in there, bird.
So let's, whatever is available we can put cat in here.
I'm asking ml5 to load the cat model.
It's actually doing this from the cloud,
the model, all of the data for this machinery model
is stored on a server somewhere,
and ml5 knows how to load that for you.
So, then what I'm going to do is add a callback,
I'm going to say model ready.
I'm going to write a function called model ready
and I'm just going to say console.log model ready.
So let's just run this code and see if it even just works
loading the model without an error.
So I'm going to close this demo down,
we have my own here.
Okay, great the model is loaded,
and just to be sure,
oh yeah initialized SketchRNN model ready,
just to be sure if I put in here like tiger,
which I don't think is something in there.
Oh, maybe it is (laughs)
guacamole, that must not be something in there.
I spelled guacamole wrong even.
Okay, right it couldn't even find guacamole
and I know that's spelled wrong,
apologies to guacamole lovers all over the world.
Okay, so we've got the cat now what do I need to do?
The first thing I need to do as soon as the model is ready
is ask for something.
So what SketchRNN will give you
every time you ask for it to generate something
is what is referred to as a stroke.
Now, I'm not going to use the variable name stroke
in my code because stroke is a global function in p5
that sets the outline color of a shape,
and I don't want to get confused with that,
but the stroke is a JavaScript object with a few properties.
It has dx, dy, and I think it's just called pen.
And dx is a floating point number, some number like 1.3,
dy is some number like negative four,
and pen is a string.
It can either be up, down,
or end.
And what this refers to is a particular
stroke of a pen basically.
So if you could imagine dx for the change in x, delta x,
dy for the change in y, delta y.
If I were to go over 1.3 units and up negative four units,
basically the stroke is this.
This is the path of the pen,
that's what SketchRNN should draw right now,
and it should either draw it as a line if the pen is down.
If the pen is up it should just move
from here to there, right.
Because if I'm drawing a cat, I'm going to draw this,
then I'm going to pick my pen up, move over here,
so that action is also a stroke
but it's a stroke with the pen up.
And then pen end is when the drawing is finished.
So this is what ml5 does, the actual native SketchRNN model,
just gives you all these things as array of numbers,
and so we've kind of made it as a JavaScript object
that makes it a little bit easier to read hopefully.
Okay, so now coming back over here
I think I just say model.generate
got sketch (laughs).
Oh, oh, oh, yes!
Now actually, an important thing that I should probably do,
I don't necessarily need to do this the first time,
but because this is a kind of machine learning model
that is giving us sequential information
the next stroke, like the one that comes after this one
is quite important, it's related to this one.
But if I want to start over and draw a new cat
I need to kind of go back to the beginning
and say start a new cat over.
And so a model.reset function is the thing that does that.
So model.reset I'm going to call that,
It should be reset the first time you load it,
but I'm going to call that anyway.
Then I can say model.generate got sketch
and then I'm going to write a function.
I'm going to call it got sketch.
It should get that stroke path.
Let's just call that S and let's have a variable.
I'm going to have a global variable called stroke path.
And I'm going to set that I just going to
set it to null when the program starts,
and then when I get a new sketch I'm going to say
stroke path equals S, and then I'm going to say
console.log stroke path.
So let's see if we get this first thing.
Model loaded!
Undefined oh.
(laughs)
That was close!
(sad trumpet sound)
I forgot, this is actually very hard for me,
I always make this mistake, ml5 is written in such a way
that uses something called error first callbacks,
this is different than how p5 is written.
So this got sketch function is receiving,
what was actually undefined there was the error,
there was no error.
All the callbacks need to have an error argument first.
So, if I wanted to be really thorough about error handling
I might do something like this.
So I could console out the error,
and then if there is no error
now just console log out the stroke path.
So this should work now.
Load that model!
Ah there it goes!
So look at this, I need to move
this many pixels in the x direction,
this many pixels in the y direction, and the pen is down.
So this should be easy enough for a sample then.
If I have an x and a y global variable,
I'm going to start them in the middle.
Then I am going to, in the draw loop I'm going to say
if stroke path is not equal to null, right.
I got to check do I have a new stroke path, right.
The new stroke path will come in the callback here.
And draw is always looping, so draw's looping
to wait to see that there's a stroke path.
And again I could control how the draw loop works
with the query to the model in a different way,
but this is like an easy way for me to do it.
Draw's just going to loop, then I'm going to say
draw a line from x, y, to x + dx, y plus dy, right.
Just draw that line and then what?
Let's set stroke path to null again.
So once I'm done set that stroke path to null.
Okay, so we should just see that one first line.
Oh dx is not defined.
Strokepath.dx, right it's part of stroke path.
I don't know why, the dx and dy is in the object.
Here it goes.
Do you see it?
Is there a line?
How come I didn't see that line?
Oh, hmm.
Stroke zero?
Stroke weight,
stroke weight four?
What did I miss?
Line...
Oh you know what!
(laughs) I'm sort of forgetting.
I just don't want to, I just want to draw the background.
I'm redrawing the background, so as soon as I draw the line
(bell ringing)
that's such a mistake that I always make.
I drew that line but then draw looped again
and drew the background over it.
So this is a thing where I just want to draw
the background once and I'll draw it in setup in this case.
So let's draw it at setup.
Here we go, we're getting there.
Let's see that line.
There it is!
That's the first stroke of our cat.
So we got the first path that the neural network
the SketchRNN model imagined for the cat.
So now what we're going to do is...
Ah wait what do we need to do?
Are you thinking what I'm thinking?
Let's just ask for the next one.
So how did I ask for the first one?
Model.generate, I can just say okay I got one,
set it to null, generate and wait for the next one.
Also though I need to move x and y to that new point.
So it actually might make sense for me to say
like new x equals, and this will clean up the code
a little bit in a way that makes it more readable,
this is the new x value, this is the new y value,
draw the line from old x to new x,
set the stroke path to null, ask for a new one,
and also move x and y to the new position.
So now when there's a new one we should get the new one in.
Guess what?
Here we go.
Let's see our cat.
There's our cat!
Whoa woo-hoo!
And it's console logging its drawing,
now, that looks crazy, right?
Because I completely ignored the whole
pen up versus down thing.
So that's something really important.
And there's something that actually I didn't bother
to tell you that's kind of important
and makes it a little bit more confusing.
Let's take out this console.log by the way.
So, the thing that's extra confusing about the pen up
and pen down is when you get this information
it's actually telling you the pen state for the next stroke.
This is not the pen's state for this particular path,
it's the pen state for the next one.
Why is that?
Well, it's a little bit weird but if you think about it
when you first start drawing the pen is down,
it has to be down, that's the definition
of the beginning of the drawing.
And then when you're done you're saying
finish that last one then end.
So, what it really is it's the pen state
that you get in the previous stroke sets the next one,
and when it's end we're done.
So let's try implementing that, and a way we can do that
I think is just by having a variable
that's called like previous pen, and previous pen,
or I can just call it pen actually, let's just call it pen.
Because by definition pen is down.
So pen is going to be down to start with.
Pen is down.
And here I'm only going to actually draw it if pen,
whoops sorry,
if pen equals down then I should actually draw the drawing,
draw the line sorry,
I should always, oh I should always move X, right.
So I always want to move X, but I only want to draw the line
if pen is down, and then what I need to do
is here I need to say pen equals,
oh I am going to need like a previous pen.
Or can I do this like right before
I get the new stroke path?
I'm confused.
Oh I can just say, I can say pen equals strokepath.pen.
So basically the pen starts as down,
then I'm going to draw and then pick up that
for when the next time comes around.
So don't pick up the actual pen value from the object,
you might need to think about this for a little bit.
This is really like previous pen,
maybe I should call it previous.
Because really what I'm checking here
is the pen from the last time
'cause I'm picking it up after.
Let's see if this works.
(snare drumrolling)
Look at that cat! (laughs)
I was waiting for like, here's the thing,
we don't know what we're going to get.
This has just been trained on what
users all over the world drew cats.
And sometimes we're going to get a nose and whiskers
and sometimes we're not.
We're going to see a variety of different cats.
Okay, but now we also need to figure out what to do,
we need to check for pen ending.
So, if I here were to say
if pen, let's set the stroke path to null,
we can do all this but as long as pen is not equal to end
then I am going to say generate the new.
So if pen is equal to N don't bother to generate
another stroke, and then I'm going to just put in here
else console.log
drawing complete, okay.
Let's give this a try.
We're going to see the cat.
There's out cat, there's our cat.
Drawing complete!
So maybe we should have it restart a new one?
And this is going to be good for our snowflakes,
you'll see in a second.
So actually instead of drawing complete
what I could actually do now is I could say model.reset
and model.generate again.
So this is going to instantaneously do another one.
Let's see if this works. (laughing)
Here's a cat, oh you know what,
it's going to do it from wherever it ended.
So the other thing I need to do is reset where x and y go,
so let's say x goes back to the middle of the screen,
y goes back to the middle of the screen also.
It takes a while to load the model, so each time
I'm refreshing I have to wait a little bit. (laughing)
So let's see our cat, I also need to give myself--
oh look at that it's a very different kind of cat.
Drawing complete now it's doing another one.
So this is perfect because this is exactly
what I want to do with snowflakes,
this is exactly what I want to do with snowflakes,
so first let's change this to snowflake.
Let's make this x a random place.
Let's make this y a random place,
and actually let's also translate to the center.
I want to translate to the center,
and I'll explain why in a second.
So actually this is going to be at random,
anywhere randomly in the canvas,
so x is going to be between negative half width
and positive half width.
Y is negative half height, positive half height.
And then I want to do exactly the same thing
whenever I finish the snowflake.
Okay, here we go.
And let's see what happens.
I don't know how big these snowflakes are going to be.
There's a nice snowflake.
There we go, look it's drawing snowflakes. (laughing)
So I wanted to create a big scene of snowflakes,
so I think in order to do that
I need to scale everything down.
And so I could do this in so many--
what if I just...
What if I just did this.
What if I just scaled all the dx's and dy's
by like one percent, by ten percent.
I was actually going to do it a totally different way,
but would this actually work this might be
an easier way to do it.
Yeah, there we go.
All right, here is our festive world of snowflakes,
and I am going to make this larger,
I'm going to get rid of the console, do this.
Whoops.
(laughs)
Nice big canvas.
Oh, there we go that's good enough.
Okay, go!
Draw snowflakes!
So (laughs) there we are everybody.
So you could imagine you could probably create
a whole scene with a bunch of different things,
and do much more.
I'm going to let this run for a little bit,
we'll speed this up so we can see what
this scene of snowflakes looks like in a couple minutes.
(light cheerful music)
All right, so as you can see--
by the way during all that time (laughs)
I changed the background to black, made it sort of
full screen and drawing the snowflakes white.
But you can see this is what after it's drawn
a whole bunch of them this is what we get.
So I would encourage you, this is what's going to be
interesting about this is oh could you save those in objects
and then combine this with my last year's
snowflake snowfall coding challenge?
Could you think about color, could you animate them
in some way, what other types of things might you do
with SketchRNN into a scene, so many possibilities.
I'm sure you will create something.
Take a look in this video subscription
for the link to the codingtrain.com page
where you can submit your versions of this,
and hopefully when you're watching this
there'll be more information also
about SketchRNN on the ml5 website itself.
Okay, thanks for watching!
(tooting train whistle)
(light cheerful music)