what's going on everybody and welcome to
a new series on hacking around with
neural networks just kind of doing
unconventional things with neural
networks and the type of model that
we're gonna start off with is the
generative model so what I have here is
a simple program basically where I just
ask the model hey draw me a number of 0
through 9 I'll pick the number I'll say
please draw me a 5 for example and that
5 becomes a sort of primer for the
generative model the generative models
then generates me something like a 5 now
the interesting thing about this model
though is that first of all this 5 is a
totally unique new novel 5 it's never
it's not in the data set we've never
seen it before and if I ask you to draw
me yet another 5 what it's going to do
is draw me yet another unique new novel
we haven't seen it before 5 and so so
the reason why I think generative models
are interesting is one they're still
like a real like area of research and a
lot of people in the fields of deep
learning kind of write off generative
models I'm drawing a 4 by the way just
so you know
so for anyways the people in the fields
of machine learning tend to write off
generative models because they don't
quite fit our paradigms today of
problems that we'd like to see solved so
classifiers on the other hand with deep
learning there's kind of two ways that
classifiers with deep learning have
improved and that's with incrementally
better accuracy and results and then
also with and there's our nice beautiful
4 you get the idea and then also with
solving problems where there's like
millions of samples that we just could
not train with support vector machines
but arguably I would say those results
are kind of based more on just using
GPUs generally so GP GPUs and so those
two things kind of happened at the same
time so anyways so incrementally better
results and then able to solve some
classification problems that we've never
had before but with generative models
there
it just opens up an entire new door and
so just because we can't think of use
cases for them doesn't mean we need to
write them off immediately I think
probably part of the problem is a lot of
times funding for research and stuff is
done by companies and stuff like that
that that have a use case in mind and I
think that's why we see these as being
useless but anyways
no one's funding me besides you guys so
Python program internet social support
and so anyways we get to play around and
do fun things so with that let's get
into the the tutorial where we're going
to you know get to this point but also
do some other playing around with other
things and just kind of poke around so
what we've got what I'm working with is
Python 3.6 tensorflow
1.7 and then we're also going to at
least start off using the following
package so hello here so the the commit
for this exactly is 401 e BFD it doesn't
look like it's too actively maintained
so maybe it won't be a problem but if
anything's not working you can get the
exam python version same tensorflow
and same same package that we're going
to use here so go ahead and clone
download that and oops I somehow got a
copy of it and then what we're gonna do
is we'll break out of this one move that
aside and once you have the package
we'll go ahead and extract it and let's
just go over what this is so this is a
character level generative model and
what's gonna do is you just give it any
characters and then what's gonna do is
you're gonna say okay with all these
characters I want you to train on it and
then be able to generate me something
similar to what what you've seen so it's
it's like a classifier in the sense that
you can produce you know there's an
input and there's an output and the goal
is to fit the output to the input based
on the training data so it's very
similar to classifiers in that sense but
the difference is a generative model can
take a variable number of input and it
can produce a variable sized output
and with tensorflow recurrent we do have
these dynamic recurrent layers which are
slowly becoming capable of doing this
sort of thing but not quite the same so
I think the best way to to show you guys
what generative models do is to just
kind of use one and work with it so so
this is the package we can check out
trained up PI just to kind of see what
what's going on but basically you're
gonna pass in you know you're an
astronaut PI it's gonna look in your
data derf or something but we can all we
can use the command-line arguments to
pass some other form of data we're gonna
ask it to you know we're gonna tell it
where to save where to put the tensor
board blogs basically the size of the
network all that stuff but there's
there's defaults for all of this so you
can kind of look through that once you
start getting past a sequence length
maybe number of epochs you probably are
done changing things I wouldn't suggest
you mess with these until way down the
line but the main things to look out for
or the the size of the model may be
batch size if you make a model too big
that it can't fit in the memory you
could decrease batch size if it's not
too big you could increase batch size to
use more of your you know utilize more
of your GPU possibly what was the other
thing
Oh sequence length so how long have
sequences right now it's gonna look at
the last 50 characters each time it
wants to generate one new character it's
gonna look back 50 characters so that's
what's gonna do also we can see that
there's maybe some some starting data so
let's check out data slash tiny
Shakespeare and basically we're looking
for input text so let's get let's go
data a tiny Shakespeare input text and
here's what we have so as we can see
this is some Shakespeare so it's a play
basically and and there's a few things
here though that you want to kind of
pick up one is is you see there is some
structure to it you've got basically
what appears to be the structure of name
: new line sentence new line new line
name : new line sentence new line new
line and so on
so we have that also we can see that you
know basically you've got name which
should just be like one or two words
most likely then the sentences could be
highly variable and then it's like Old
English basically but it's clearly
English ok so yeah so that's our
training set the other thing that I
really think is cool about generative
models is the size of data that's
required this is only one megabyte of
data that's it
so so the fact that if it works
spoiler alert it works if you haven't
seen enough spoilers already that's
really cool that you can do this with
such a small amount of data so anyways
so that's our starting data set let's go
ahead and train the model so coming back
into this main door where we've got
trained up high let's go ahead and
actually just run Python train Pi so
it's just going to use all the defaults
we're gonna change them don't worry but
slowly entering into the world of
generative models so let's make sure
this actually starts training and once
it does I'll just pause it and then I'll
unpause okay so there it goes if yours
is not training this fast you're
probably on a CPU or maybe a lesser GPU
but I'm pretty sure that this doesn't
even come close to fully utilizing my
GPU like I'm recording on it and my
desktop is totally fine I can see my
mouse moving pretty pretty smoothly
anyways what I'm gonna go ahead do is
just pause the recording while this is
running and then I'll pick back up when
we've done I don't think I'm gonna
finish all the epochs but I'd like to
get us do at least a few so anyways I'm
gonna pause it while this is going and
I'll pick back up in a few minutes but
I'm not gonna stand here in total
silence with you all okay I actually
found that it rained pretty quickly so
not a problem now one thing that we can
do is we can always check to see how
training is going and kind of make a
determination whether we need to
continue training or not so you can
always run
oops I'm off screen
it's just tensor board - - log
I think it's log underscore deter I can
never remember no it's one word log dirt
equals and then we'll just say logs
because that's where the logs are then
what we can do is we can navigate to
this URL here yours will be a little
different probably not HPC and here is
our training so actually you know to me
it looks like you probably could even
kept going and probably made it a little
bit more progress but that's okay so so
looks like it trained let's see what it
learned so the first thing that we're
gonna do is we'll quit out of here and
now we want to run sample dot pie so
with sampled up pie we actually can just
run straight sample dot pie but it also
has command-line arguments that we could
pass so you would say where's the save
directory you can prime it in this case
it's priming it with a space and then we
connect and then here we have the actual
sample here but I'm a prime not going to
touch that but but the big one too is n
so this is how many characters do we
want a sample so it'll just continue
going and in this case it's going to
generate 500 samples so let's go ahead
and pull back up where we were and let's
go ahead and python sample dot pi and
we'll just go with the defaults cuz it's
it's already set up to work with this
data set but later on we'll change some
things so so here's here's the the
return that we got and you'll see that
it's got you know the the B here and the
new lines aren't actually making new
lines so one thing that we could do this
is encoded for utf-8 so one thing we
could do is just not encode for for
utf-8 or we could just decode so for
example we could say like later on down
the road I think probably the best way
to do this is going to be actually
something more like data data equals
model and
you can you know do whatever you want to
do with that data but then if you wanted
you could just print data decode from
utf-8 but keep in mind that you know in
this case we don't want it to be utf-8
because it's really just ASCII
characters and new lines there's really
no fancy characters but if you're using
fancy characters like if you're you've
got like you know Chinese or something
like that with the the characters you're
probably gonna want to keep you know
utf-8 anyways let's run it one more time
just with sample that PI and then
hopefully we'll get to see the actual
structure a little better than what
we're seeing here right so as you can
see wow we even got Romeo that's the
first time left ah that's actually kind
of interesting I've never seen it
maintained usually makes up new names
Queens also there I don't think I've
ever seen it put Romeo though that's
cool anyway Pam
let's do N equals let's do a thousand
let's just do a little bit more than
what we have cool anyway so what we can
see is it's clearly learned the whole
you know name : new line all that stuff
and then in repeat one thing I find
interesting though is its capitalizing
the names let's go back into the data
set real quick
interesting so at the beginning of this
data set the the names are not
necessarily capitalized probably cuz
they're not names like it's like first
it's like not unknown entity I guess but
then once we learn once we have a name
it is all caps I guess that's just part
of the structure interesting yeah but if
you actually know the person's name
it'll be in all caps so that I guess
that's why that was an all caps I wasn't
I was wondering why was it doing that
yeah but you know so not only didn't
learn the structure it also learned I
mean some of these things aren't quite
right but you draw the Lord bount cursed
these are words bount isn't really but
you know
my throwings at home by vis not a word
you know obviously there's like some
words not here also it's not really
totally coherent but at a quick glance
you would say hey yeah that's definitely
what we trained on so now the next thing
I'm curious about when when I first saw
this was could we if it can learn this
structure right this is pretty basic
structure in English could we get it to
learn to code like could we get it to
learn to do this on Python so that's
what we're gonna do in the next tutorial
is we're going to see if we can get a
generative model to actually learn
Python instead of plays so stay tuned to
the next video if you like this content
you like what I'm doing on a channel you
go to a Python program and then it slash
support support what we're doing here
otherwise I'll see you in another
tutorial