what is going on everybody welcome to
part eight of our chat pot with Python
in tensorflow tutorial series in this
tutorial what I'd like to do is talk
about some of the more high-level
concepts and parameters of our chat bot
with the neural machine translation code
that we're using and I hope to at least
give you an idea of a better idea of
what's actually going on here because
there's actually a lot going on here my
initial intention was to start with the
basic sequence to sequence the old
English to French translation model that
was done with tensorflow but tensorflow
deprecated it and you really can't run
it without running the like tensorflow
1.0 and it leaves at the time of my
filming this we're already at 1.4 and so
I just was like yeah well I'll just
start with with the current nmt model
but it's gonna be a lot of information
that I'm about to throw at you so I
apologized here we go so so first of all
anytime you've got a translation whether
it's you know neural machine translation
translation from one language to another
language or in our case even though it's
English to English it's still a form of
translation that's gonna be going on
there it's just input to an output of
language anytime you have that obviously
words are not numbers so the first thing
that we're gonna we need to do is we
need to tokenize inputs and an easy way
you could tokenize inputs is just a
split by space in probably punctuation
okay so I am a student tokenized there's
a special token for I a token for a
space a token for am token for space
token for a and so on so you you
tokenize that input and generally that's
what the end coder is gonna do is just
gonna be tokenizing your input and then
once it's tokenized
the other thing an encoder is likely to
do is take those tokens and and one
thing we could do is just assign it like
an arbitrary ID but that's not ideal
we'd like those IDs to be somewhat
meaningful so we generally are gonna
create a word vector that is gonna put
similar words and give them similar id's
and this is going to help both in
actually translating but also it's going
to help us in a valley
waiting how good our translations were
because a lot of times you you might be
really close to like like translating
you know let's say you've got airplane
in Japanese and you translate it to a
car or or better yet air yeah airplane
in Japanese you translate it to car
versus airplane in Japanese and you
translate it to I don't know
shoe or cat or something like that right
at least car is closer to plane than
shoe or cat is so we would know that as
we get closer that that correct
translation we want a reward for that we
don't want to have necessarily equally
wrong just because we miss the word
right so generally we're gonna have word
vectors as well once you have those IDs
though then you're gonna fit through
some sort of a neural network with
language information this is generally a
recurrent neural network that just makes
sense so we have that sort of non static
temporal sense going forward and then
generally it's a recurrent neural
network we're usually using long
short-term memory at lsdm then once
you've done that you feed it again
through a decoder and then you've got
your output and that's your basic
sequence to sequence on language data
translation model now let's start
talking about some of the problems that
we might have while doing that so first
of all your input and your output
first of all these don't even match
right I am a student is for tokens not
including spaces and then the output is
three tokens again not including spaces
okay so so already those don't match but
also I a Mista is your input always
going to be those four tokens or what
however many tokens you're gonna have no
right your input is gonna vary right so
so at least the way like initially that
we solve for that was with padding so we
might say okay what's our longest
sentence we've ever had let's say it's a
hundred tokens so the longest sentence
is a hundred tokens let's say we've got
we just got a sentence that's five
tokens but what we're gonna say is the
input layer is always going to be a
hundred tokens or a hundred nodes and
each nodes value starting value is just
going to be the ID hopefully a
meaningful
it's gonna be the ID and then let's say
we only have five tokens well we put in
those first five and then we have a
special token called the pad token and
then we're just gonna use that pad token
on every single node after so we just do
a bunch of padding and that's one way we
can solve it but of course that's not a
good idea most likely and it doesn't
train well and it doesn't perform well
because what ends up happening is
because we use padding so much the
longer your sentence the less impact
those later words are gonna wind up
having because the neural networks gonna
learn like okay these pads don't mean
anything so we're just gonna change the
weights and then we get a long sentence
once in a while in those last few words
just don't mean anything it just doesn't
train well to do that so then again
still on that first sequence to sequence
translation and again that was also this
is char like if you guys have been
following either the twitch stream or
you follow Charles the AI on Twitter
that was this is what he has been
running so he's been on that kind of v1
initial sequence a sequence model from
tensorflow
until very recently until this series
started basically so so the other idea
was that we could use bucketing so the
idea here was like okay well what we
could do is we could have input layers
that are buckets so we'll take our
tokens and we'll have like four buckets
we'll have a bucket for the stuff that's
like five to ten long we'll have a
bucket for the stuff that's eleven to
twenty five long a bucket for twenty
five to thirty five and then a bucket
for thirty five to fifty or something
like that we would do bucketing and then
whichever bucket held the longest
version of that string that's the bucket
that we would use we may be trained and
this did okay
dude oh it did alright but not ideal
because we still are going to be using
padding it's still at so that brings us
to today with tensorflow we have what
are called dynamic recurrent neural
networks and we can have a dynamic input
coming in now moving back in time a
little bit and also getting into now
again now we're going to be talking so
what I've kind of talked about up to
this point right before dynamic was that
first sequence of sequence stuff
tensorflow and now what I'm gonna be
talking about is actually the nmt the
current age and MT that we're using so
another problem that we have besides the
bucketing which is solved besides the
bucketing and padding which is basically
solved by dynamic recurrent neural
networks we have the issue of just
language in general so for example let's
let's consider a translation not a
chatbot but a translation task like
English to French in general if you want
to translate English to French or
English to German or English to Spanish
there's there might be slight variance
in the syntax of like a noun phrase but
in general there's a algorithmic solving
of the translation of English to French
German Spanish whatever that's pretty
pretty simple I guess and it goes in a
linear order but then you have a
language like Japanese so English and
Japanese are very very different
languages and they don't follow like
even remotely similar rules and in
Japanese sometimes that last character
changes the meaning of all the other
characters and and things are just
totally different right and then also
with a chatbot the same thing is true
and and in a lot of translations the
same thing is true that it's like with
an LS TM generally an Alice TM is really
pretty good at remembering in a sequence
about like 10 to maybe 20 tokens right
so let's say tokens are words so in our
case we are tokenizing to word another
option you can do - tokenize is to
tokenize by character so each little
character ABCD and so on or you can
tokenize by like kind of like almost dub
try to think of the word oh I can't
think of this stupid word syllable
anyway I don't know I'm totally blanking
my brain is done anyways I think it's
syllable right just chunks of letters
basically that hat like hair is Sun or
hair is Sun right you could tokenize by
those kind of little bits and you'll
have much less and that's bpe
tokenization and we might be talking
about that later on
but for now we're tokenizing by word so
anyway think of it by words so think of
like
how hard it would be for you if you
could only remember you if you could
only respond to 10 to 20 tokens max at a
time in really 10 okay so you needed but
you think about if you needed to build
your response to ten tokens at a time
and build out your response as you slid
a window of ten tokens at a time so how
hard it would be if you write or think
about how hard it would be if you write
that okay start building a response you
don't even know if you what right and
then how hard it would be if you could
only or something like that right keep
sliding it and you have to build a
response the neural network must
generate a response so this can be very
challenging if we're only thinking back
historically but also only if we can
only remember 10 to 20 tokens the other
issue is how hard it would be okay think
about how hard it would be and then if
you could only and that that's kind of
like almost like a filler if you could
only and then think in terms of 10 to 20
tokens at a time it's it's almost like
the first part in order to understand
that first phrase you have to get that
last bit too but you also couldn't take
just the last bit if you could only
think in terms of 10 to 20 tokens right
you know now you don't know how to
respond to that if you could only think
into what what do you mean you have to
go back to the think about how it would
be right and so this is where we have
kind of two new concepts that are coming
in here which is both at least in terms
of using future data as well with
bi-directional recurrent neural networks
so in a bi-directional recurrent neural
network we're going to feed data both
sequentially forward and then also kind
of reverse order backwards through that
hidden layer in the model so that's one
thing and then we're also going to make
use of attention models which you can
again tension models are in this
tutorial there's also a great paper on
them so this kind of explains attention
models a little bit and then I also took
an image from the the paper on attention
models which I think pretty much drives
the entire point home so this is the a
graph of blue score to sentence length
so think of these as like your tokens
and this red line here is base
no use of an attention model and a blue
score is bilingual evaluation under
study I think it's basically it's just a
score of how good a translation was
basically so so the more the better
and as you can see like the model is
pretty good at ten even at like just
before twenty it does its best blue
score but then a very quickly soup just
falls off especially as you get to
longer sentences which is problematic we
we tend to speak in pretty long
sentences and sometimes you need
multiple sentences to understand a
meaning so then these are just with with
blue score or some attention models
being applied and in the real thing to
drive home here is it helps both on the
very short stuff but also on the longer
tokens it's still like it basically
flatlines after like 40-ish there's a
slight decline there but it pretty much
holds out all the way up to seventy and
probably out further from there so so
the attention model is gonna help us
remember longer sequences at a time
which can help us to kind of brute force
our way through this kind of context
problem where we need both historical
and future information but then we use
bi-directional recurrent neural networks
to kind of mix it up and because
language is not necessarily only in
perfect sequence sometimes we have to
use context and we have to hear the full
sentence before we can go back and
respond to words that we heard leading
up to that point so real quick what I'd
like to do is just is visualize a
bi-directional recurrent neural network
so let's go do that really quick on a
simple recurrent neural network you have
your input layer your output layer and
then we'll have one hand layer for
simplicity sake then your connections go
from the input layer to the hidden layer
where each node in the hidden layer also
passes data down to the next hidden
layer node which is how we get our
temporal and not so static
characteristics from recurrent neural
networks as the previous inputs are
allowed to carry forward to some degree
down on through that hidden layer on a
bi-directional recurrent neural network
basically the hidden layer has data that
goes both down and up through or bisily
in both directions through that that
hidden layer so you still have your
input
you're still gonna have your output
layer you're still gonna have the
connections from the input to the hidden
and from the hidden to the output but
then also in that hidden layer you've
got basically the data that goes in this
in this drawing down and then up or
forward and then reverse depending on
what what what drawing you're looking at
and then from here you're gonna have in
theory this is actually fully connected
just because again this is all that
hidden layer is really there's just one
hidden layer all those nodes are part of
the same hidden layer but then you might
also see more fancy types of connections
just to just to get it more than just
simply forward and reverse just get a
little bit more complexity out of the
network while we're there but anyways
that's the difference between a simple
recurrent neural network and a
bi-directional recurrent neural network
because in a lot of tasks it's not just
what happened leading up to a certain
point we do actually care what happens
after that point as well so it's still
important for us to could actually go
both ways all right so now what I want
to do is cover some of the training
metrics that you're gonna come across
and give you an idea of kind of what at
least I've found to be useful as far as
training the model and kind of tweaking
it as time goes on and and kind of
knowing when it's done so the first
thing I want to do is show you tensor
board hopefully you're familiar with
tensor board already but if not here's
here's what you can do to run it
basically it's just a way you can
visualize how training has been out
train is going basically right now you
can visualize a lot of scalars and
there's some other things that we can do
and I'll show you one of them so first
of all to run tensor board you can run
this while your algorithm is training so
what you want to do is go into the model
directory don't worry if you don't have
all these same directories I've made
lots of copies and did lots of things as
I'm training so anyway head into model
and then train log is where you have all
your logging files you'll probably just
have event files like this you won't
have necessarily these other files so
from model what I'm gonna do is I'm
going to open up a command prompt just
with CMD typing it in there and then
just type tensor board - - log door
equals train underscore log
and then you would hit enter now
actually I went ahead and cancel X
actually I already have it up I wanted
to bring it up prior because it can take
a little bit to load and I'm loading
like a hundred thousand steps so anyway
it's just tents aboard log door train
log now once you have it up this over
this is my tensor board for Charles v2
so he was trained with I think about
three million pairs and then right now
I'm training a model with like 70
million pairs so hopefully it'll be
better than this one but anyway this is
kinda this is this the the tensor board
and the information that I had so the
big things you want to pay attention to
are the blue score blue is blues
probably the best determining factor in
at how good a translation was the
problem is we're not really doing
translations so so basically with when
you're translating English to French for
example in general there's either just
one proper translation or maybe there's
three or five but when you're
translating English to English when
you're doing like a chatbot like an
input to an output a comment in a
response from any given comment it's
really like infinite responses there's
there's no limit to what the response is
and it's and unless it's not coherent
and unless it doesn't make any sense at
all it's a valid response so so for me
the blue score is you know relatively
useful we'd like to see it go up but I
don't think we're gonna see a blue score
of like 20 okay
we're probably gonna see blue scores
around 3 4 max like there's really no
reason why we would see a super high
blue score unless maybe we over fit too
much like if you if you train on like I
don't know you did like 5,000 epochs or
something like that right you'd probably
see a pretty good blue score but you
probably over fit so so yeah I wouldn't
I want to put too much weight into that
but in general we'd like the blue score
to go up okay
slipped gradient I wouldn't I'm not
really paying much attention at and then
the gradient norm the other thing I
I think in the previous tutorial there's
like GN I'm not again I'm not positive
that's gradient noise but I'm gonna go
ahead and guess that after real looking
at this it spread gradient norm that's
my guess anyway initially you want that
gradient norm to kind of fall but
honestly Ivan I just haven't paid much
attention to it
then we've got the learning rate as you
can see here I started with 1e negative
3
after about wonderful epoch I went ahead
and dropped that and I basically just
have it every time so there's a bit you
know there's like basically one half
here half and then one final half so
from you know zero point zero zero one
to zero point zero zero zero five and
then to five and then 0.0001 hope I got
that right for you anyway you just keep
having it and basically 1e negative four
and then once you've got to 1e negative
for like you're not gonna get any better
than that oh you're it's just not gonna
work out for you to keep having in my
opinion and that's what the atom
optimizer again if you're gonna use like
stochastic gradient descent then you'd
probably be better off doing some sort
of auto decay kind of structure and that
makes more sense but with atom the ad a
literally stands for adaptive and then
the M is momentum I think anyway it's
adaptive so you don't really need to be
messing around with your learning rate
as much as you would with like
stochastic gradient descent where you
might decay every step or every 100
steps or whatever in general I would
really just decay atom maybe every one
to two epochs and if you want you can
calculate that out batch size is 128 so
divide how many samples you have by 128
that's how many steps it will take for
you to go through a holy book celeste
learning rate train loss hopefully
you're not you know what train loss is
but generally obviously we'd like to see
that going down and basically I stopped
the model because one I've I had already
decreased the learning rate as much as I
felt was necessary and it looked like
train loss was maybe even starting to
climb again so there's really no reason
to keep training this model in my
opinion and then for whatever reason
when we do a bi-directional recurrent
work this graph just doesn't work so I
think that's a bug in the in the code
because it does work totally fine with a
non bi-directional recurrent neural
network anyway that's your tensor board
stuff I do want to bring up
I guess I'll bring down the other one so
I have you have a model that's training
right now on paper space let's do that
okay I think you can see that yeah so as
you can see this is tensor Road I got
sensor board running right here I'm
currently on step 59,000 things are
looking pretty good I haven't decayed
anything just yet and train loss keeps
going down the other thing I want to
bring up too is so so there's two to two
major metrics right blue which we've
already discussed it's basically how
good of a translation is it and then ppl
is perplexity and perplexities is this
it's basically just a probability
distribution so it's how far off are you
so blue score it's like the more the
better
whereas perplexity is we'd like that to
be as small as possible
we'd like perplexity to be in like the
single digits if possible again on
English on a chatbot that's probably not
gonna happen on English to French we
actually could get a perplexity in the
single digits but yeah probably the
chatbot we're not gonna be able to do
that and if we did we probably just over
fit that's my guess so anyway the main
numbers you're looking for is perplexity
in general if I recall right the
perplexity on Charles V to the one
that's live on Twitter at least at the
time i recording this although he's
probably gonna be replaced with this
model when it's done I I think you
gotten to like the 50s I don't think he
even gotten to the 40s for perplexity
but these are just general numbers again
the hard part with English to English
with a chatbot is there's no set one
answer so you want to get generally
vaguely close but you're not going to
see incredible numbers there so moving
this aside the last thing I want to show
is the you know basically the reason for
a word vector and I've brought up I've
already kind of loaded it so it can take
some time to load so this is the
projector
okay for some reason it's looking like
it might not work and then these are
like our basically our word vectors verb
the Inc this is the encoder but you can
also see the same thing with the decoder
but anyway we can kind of zoom in and we
can kind of see the actual words
themselves on my mouse keeps loading
it's annoying anyway
eBay and yeah eBay is related to eBay no
surprise Amazon you know things you
might do on your computer wireless more
eBay USB SSD buyer yes like computer
stuff hmm some some word I've never seen
before it must have something to do with
computers setups Camaro router more
computer stuff ranging waterproof as you
can see stuff I wonder what that one
word was that's weird Walmart
photography iPhone anyway you keep
looking there's a hundred that like the
vocab here was a hundred thousand so
there's obviously a lot of words here
but anyways so and this is kind of how
it's gonna work like it move it cheetah
it's pretty packed childhood memories
sorry this is kind of fun
won't be long until we come up with a
pretty bad word probably so I approached
it stopped anyway this is a
family-friendly tutorial so I think
that's enough for now with various
metrics that you should be looking at
and all that again in general training
loss you want to see that go down once
it flatlines you might want to consider
decaying at least with the atom
optimizer you probably should stop
decaying at one negative four and then
in general we'd like to see perplexity
falling we'd like to see single-digit
perplexity but that's probably not gonna
happen with the chat but at least not
yet anyway and if it did it'd probably
be unreliable as my guest
blues score we'd like to see that go up
it's really as simple as that and
hopefully we'd like to see mini epochs
so right now the model I'm training on
paper space is with seventy million
samples because this one was only three
million samples so I did I forget how
many parks that would be
a hundred thousand times 128 which
unless my math is wrong then be like 12
is I may like for you pox I don't know
Oh
hopefully I'm not off by an order of
magnitude but I think it's about 4
epochs whereas here 1 epoch would be
like 480 thousand steps so that is a lot
of steps whereas this one only had to do
one hundred thousand steps to do a few
epochs this one's gonna have to do four
hundred eighty thousand just for one
epoch and my hope is that because all
that data is unique like every single
input data every single sample is a
unique sample my hope is that it's going
to be much more accurate and much more
interesting and unique and his responses
won't necessarily all piece of similar
so often because every single one he's
seen even up to five hundred thousand
steps will be a new unique
never-before-seen sample and ideally I'd
hope to be able to do to epochs but that
would be like a million steps and that's
gonna take even this one's gonna take I
forget I think eight days to get to
maybe a little less than eight days
about a week just to do one epoch so
yeah I'm not sure I want to do two weeks
of training okay
but honestly oh I'll just keep training
until training loss until I get to one
negative four and training loss still
won't decline and until then I'm gonna
keep training and hoping that per unless
unless perplexity starts going up
crazily or blue score starts falling or
something like that but yeah okay so a
lot of information I probably missed
some stuff if you have questions or
anything on on what you're seeing here
do feel free to ask if you've got a
correction if I said something wrong
which I'm sure I did somewhere in here
there's so much stuff feel free to
correct me if you've got more
information you want to share with
everybody feel free to comment about
that in the next tutorial what I'd like
to do is start talking about okay well
let's say you get to the point like
right now when this model trains it
outputs files so you can go the to the
testing file so if you go into model let
me just do a refresh here yeah okay so
so you it'll output to output dev and
output test you can pull up output dev
and kind of see what his responses have
been recently so again this is kind of a
newish
model but you can see again these are
this is after a decoder but it hasn't
been fully anything of the proper word D
tokenized I guess anyway you can see
although it's probably pretty small for
you huh I don't know I think this one I
can zoom sorry anyway you can see the
responses and then I've got a script
that basically pairs the two together so
I because I wanted to be able to see it
what are these responses to and stuff
like that so I'll talk a little bit
about that how you can pair these
together and kind of see how good is it
looking so far obviously there's a lot
of hunks there's a lot of repetition in
here I'm not sure as being said a lot
I'm not happy with this yet it's got a
long way to go though as I know it's
like only 10% through a full epoch yet I
believe a little bit more than 10% so
not not too bad
that's pretty much expected but anyways
you can kind of see how it's doing like
this but eventually you're gonna get the
urge to speak to your chat bot so we're
going to talk a little bit more in the
next tutorial about how do we pair the
output dev maybe next tutorials anyway
how do we pair the output dev to the
actual testing input that that generated
this so you can kind of see are these
responses coherent or do they make no
sense at all and then also eventually we
need to talk about how do we actually
deploy the model let's say you've got
the model you're happy with it how do we
deploy it how do you do something like
launch Charles the AI on Twitter and
just in case anybody's not familiar
charles dai i pull him up here is what a
wonderful little guy anyway
this is him i give them a phone anyway
that's all for now questions comments
concerns what ever feel free to leave
them below otherwise i will see you in
the next tutorial