what's going on everybody and welcome to
part 2 of our poking around with neural
networks tutorial series in this
tutorial what we're gonna do is kind of
build on the last one so the last one we
had this input data that was basically
you know Shakespeare play and we got it
we got the neural network to produce
something that looked very just like
this basically
and then we wondered could we do this
with Python code so that's what we're
gonna try in this tutorial so the first
thing we're gonna go ahead and do is in
the data directory we're gonna make a
new one so there's tiny Shakespeare here
let's go ahead and make a new one and
I'm gonna call this one PI code and then
inside PI code I'm going to add a new
file and I'm gonna call that I don't
know standard Lib compile and I'm gonna
make that dot PI yes
and then let's open it with some blaim
so first of all we need to basically
what we're gonna do to get you know to
get our sample data we have to use we
have to have some sample code and so I
think probably what better place to get
that sample code than from the standard
library so depending on your operating
system and where you've installed Python
this could all be different but if you
just do like import sis and then we just
do cysts let's do print sis dot path if
we run that we can see basically all the
places where Python believes your path
to be and all you're really looking for
is your your your Lib path so so in my
case at C Python 36 /lib and in there
and our is my standard library so site
packages is where third-party libraries
go but in this one right here that's
where like things like you know time and
all that are stored and cysts for
example so yeah I'm gonna go ahead and
pull this back down so wearing a grab
code from our Lib
directory now the way that we're gonna
do that is let's just start over we're
gonna import OS standard Lib location
and in my case again that was right here
so I'm just gonna copy paste and if
you're on Windows just make sure you
double backslash there and let's make
sure we don't have that extra quote and
then we can specify just to keep the
size decent you could say how many
maximum files you want in my case I'm
just going to say a thousand or actually
you might want to say a hundred that
would probably produce something close
to what we're looking for what we'll see
so now why are you mad at me for this
now what we want to do is we're just
gonna count will have a simple counter
here and then what we're gonna say is
with open and then we'll do the input
dot text with the attention to append it
and with the encoding as utf-8 so with
open that as F what do we want to do
what we're gonna do is for path and
directories actually path directories
files in OS dot walk and then we want to
walk through this the standards live
location so basically it's going to give
us the path all of the plausible other
directories and then the actual files in
there and that's basically what OS
walked us it's just gonna let you you
know recursively iterate basically
through every option that we possibly
have so once we do that we're gonna just
say for file in files we're gonna say a
count plus equals one and just zoom out
a little bit maybe that zoomed in so for
file in files will count plus equals one
if that count is greater than our max
files let's just go ahead and break out
of this l if we're going to use dot pi
in file let's do a try except exception
as eve will print string either and what
we're gonna try is with open OS top F
dot joy we're gonna join the path in the
specific file so path is just the path
all the way to that file and then you
just have that actual file name itself
so we're going to join the path and the
file and we want to open that with the
intention just to read it as data F and
we're gonna say contents equals data F
dot read and then we've got that so now
we're going to input F which was yeah
we're gonna input F dot write our
contents and actually we probably called
this input F as well or we could call
this F yeah okay well just do F dot
write contents and then we'll F dot
right and we'll just throw in a newline
okay so what that's going to do is
hopefully produce us a an input file so
let's go ahead and just run it real
quick and see what happens that's fine
so at least one of them didn't work but
our input now is clearly a bunch of
Python code very well so now what we can
do is attempt to train on on this data
so what I'm going to do is come back to
our train dot PI file and basically all
we want to do here is we're just going
to change like one thing so we'll come
back to this main directory here train
dot pi will open the command prompt and
then basically we're just going to say
python train dot pi and then instead of
basically the data dirt needs to change
the data dirt equals and in this case
it's data / pi code so the data dirt
just is whatever contains your input
text so for us that
data /pi code so let's go ahead and run
this one now maybe did I not hit enter I
just want to make sure it works and then
I'll do the same thing I did before I'll
just pause it while it trains yeah so
I'm going to pause it now while that's
training and then we'll pick back up I'm
not sure I'm going to do all 50,000 nipa
or 50,000 steps rather we might pause it
a little earlier than that but we'll see
alright so we're at about 35,000 steps
here is the current training so the blue
one is the model that we're training
right now so interestingly enough the
loss went significantly low lower than
the other one which is I'm not really
sure what why why that would be cuz we
use the same parameters just different
data anyway interesting so I mean it's
it easier to learn problem so what I'm
gonna do is I'm gonna let this hit
36,000 then I'm gonna stop it and then
we're gonna sample weight food okay so
that's saved so I'm gonna cut it and
then now let's just do a Python sample
dot pie and then let's do N equals a
thousand and let's see what we've got
wait for it
cool okay so first of all this is a
little hard for us to read just in the
console like this so one option we have
is to just output it but at least right
now I mean it looks pretty good but what
we could do is we could just give it an
out so let's do 2500 and then let's just
do it to out dot txt actually we
should've done out dot pie I don't know
why I didn't do out dot pie let's just I
kind of want to stop it just do it
doesn't really matter let's just do I'm
just gonna change it to out dot pie and
now while that's running let me open up
our let's see where are we
pi code
here and then here let me shove it out
yeah there we go so now let's open that
in sublime and cool so we have some
quote issues here my own so common I
like redo it so if you like for example
I bet if we fix let's zoom out real
quick so like somewhere the quote didn't
get closed like let me let's just do
let's delete this no that still doesn't
want to do it hmm let me redo it let's
just redo to out dot PI no one looks
really messy I've seen it do much much
much better than that obviously it's not
gonna get syntax perfect also our
sequence length was only 50 so we
probably would want to make that a
little larger and we'd have a little bit
more success having only 50 is kind of a
problem this looks a little better it
looks like it never closed off it's uh
it's a docstring though which again if
part of the problem is we our sequence
length probably isn't long enough I mean
we can see that at least you know it
started to figure stuff out it looks
like it thinks it's in some sort of
class also these extra lines are kind of
confusing I think the problem is if we
we probably are having new line in
return so what I would do from here is
let's see I think we should change
I wanted to get rid of so maybe what we
can do
look let's we can just edit let's edit
sample dot pie and then here we print
data dot to code utf-8
let's also let's just do with open oh
you can't even see what I'm doing and in
fact let me just bring this easier for
me to do this in sublime so let's just
bring this up in sublime alright so here
print data data code that's fine with
open out dot pie with the intention to
write as F F dot right and we want to
write basically data dot to code utf-8
and then we're gonna do a dot replace
and it looks like we have a situation
where is this not closed on cool where
rather than just a new line we have this
like return new line so we're gonna
replace all instances of return new line
with just straight up a new line so we F
dot write that and that should be good
so now we'll save that let's come back
to where we were sampling and then we
don't want to out to out we'll just out
to the well the this script now does it
for us hopefully I don't have any errors
if we don't we'll see let me just move
this aside let's come over here
it should rewrite this file for us we'll
see if it does that I also print it out
for us
please tell me yes it did rewrite it for
us
I probably thinks that we should just to
like a longer one but anyway I think
we're gonna have a lot of issues with
like I wonder why unexpected indent and
stuff really why is that an unexpected
in then
I feel like that's valid intent and I
just I must just be stupid or so I guess
because this is not a like this is an
error so like I wonder if I did this oh
and then this needs to be probably
closed off then suddenly it goes away
okay that's weird that that would call
that unexpected in that but I guess it's
because this wasn't fully done anyway we
can see that this is looking it's clear
that it's Python code but the problem is
it's it's really really messy and and I
and then we obviously we have a lot of
instances where we've got like these
docstrings that never close off and the
reason for that is it's 50 characters is
over really quick and so I think it very
quickly forgets that it was ever in a
doc string to begin with so my line of
thinking here is that we'd actually want
to train with a sequence length that is
much longer than this the stock 50 so so
what I would suggest we do is go to the
training and instead do a much longer
sequence length now depending on your
GPU you may or may not be capable of
doing this but the following is the
command that I've already run so I'm not
going to rerun it but it was this Python
train dot pie
I changed the size of the network I
actually don't think that that helps
much in this case I didn't notice that
it necessarily helps the real thing that
seems to be helping is sequence length
okay so I actually already trained this
model if you want this model I trained
it for 97 thousand steps which is a lot
how to see whose length of 250 and I
just trained it basically overnight and
here's just like a little snippet I'm
just gonna copy and paste it but again I
have hosted this model so it's on Python
program in that it's in the text-based
version of this tutorial just go down to
the bottom and the model is there for
you so if you want to see it play with
it train it further or whatever awesome
but this is the result and or at least a
sample so as you can see it actually
could figure out oh I'm
still in a docstring and finish the
docstring interestingly enough this
looks like more than 250 characters and
maybe it was able to figure out oh I
need to close it because of like returns
and then it like realized ok I'm done
with the docstring and then closed it
because I'm that looks like more than
250 anyway this looks really good he's
like obviously there's some errors here
but uh you know that's clearly Python
like to me I just think this is super
cool because because it's definitely
learned so many things like even just
here you can tell first of all it knows
it's a built in method with this
underscore here and I'm sure you know
somewhere at least it thinks it's a
method so it's self and it's passing all
these things and this the docstring I
just think is hilarious that it did it
touch strings it knows about like I said
like clearly about like the self and
stuff like that and then you've got
unfortunately you've got some things
that are being passed to this method and
not being used which is funny and then
here like things are being used that
weren't passed but what we what we do
have is first of all it's honoring is it
honoring Pepe there - I think it is
let's see is this a yes so this is mad
because this is too long but you can see
here it's actually put the premiers on a
new line because it was gonna violate
Pepe probably anyway that's funny
but I learned all kinds of things like
define then it's a name parenthesis
oftentimes that first parameter if it's
a method is gonna be self and then : new
line and then it learned all of the
white space like just the fact that I
learned white space is pretty cool also
it learned spaces not tabs and let's see
what else if we notice like it clearly
learned like you know if statements
stuff like that which is pretty cool and
it also seems to have learned that like
if is generally has some sort of
conditional and it clearly picked up
conditionals which again is really
interesting to me and then yeah lots of
little stuff like like the doc string
that's awesome and then like the
comments and stuff like anyway I think
that's really cool and obviously in this
case you know business-wise you know I
try to think about like how can we
actually use this kind of stuff and I
mean immediately I can see like you
could probably make a pretty cool IDE
with this like if if it learns you know
as you type it's kind of like you know
like t9 text or something or not t9 uh
we're like predict when you text in it
like prediction your next words for you
and stuff like that a lot of times those
are pretty good in general those are
like Markov chains that generate that
it'd be curious to know which is a
little more accurate a Markov chain or a
generative model like this anyway it'd
be cool to have an IDE that just kind of
helps you as you code and it would be
able to predict the next things that
you're trying to do I'm pretty sure you
could do something like that with this
anyway I think that's all for now what
I'm planning to do with the series is
kind of touch on a lot of things and
then we'll probably end up coming back
to a lot of the things that we touch on
so I'm not necessarily totally done with
playing around with generating code or
just Python code but the next thing I
want to go to is doing things with the
EM nest data set so what I'd like to do
is try both classification as well as
generation with the Emnes data set and
see what we can kind of figure out so so
far what we've been doing is using a
generative model how it was kind of made
to be used and now what I'd like to do
is actually use a generative model in
ways that it wasn't necessarily intended
to be used and the reason why I want to
do this is just to show you that the
generative model is capable of doing
other things but it's it's literally any
variable input to any variable output
which is really interesting to me
obviously we have something similar with
the sequence to sequence in tensorflow
which is any sequence to any sequence
which is also really fascinating to me
and you can almost bet you are going to
see that in this series as well so don't
worry anyways that's all for now if you
liked the content you can support it
Python program internet slash so
port questions comments leave them below
if you want to chat with us there's a
link to the discord in the description
otherwise I will see you in another
tutorial