what's going on everybody welcome to
part 9 of our chat bot with Python in
tensorflow tutorial series in this
tutorial what we're talking about is how
you can interact with your chat bot so
there's quite a few ways for us to
interact in a few different reasons why
we might be interacting with our chat
bot so let's go ahead and kind of cover
all of those ways so first of all the
first way you're gonna interact with
your chat butter is that you're gonna
kind of see how your chat bot is doing
is obviously I guess in stats and tensor
board or something like that but
generally I mean our goal here is to
create a chat bot so we actually care
most about the output of our chat BOTS
the first way you're seeing output for
your chat bot is in the console as it's
training every thousand steps it's gonna
show you like source reference and then
nmt so source is where it's you know the
actual input text reference is the
training or that I'm sorry the testing
output and then the nmt is your chat bot
it's actual response that's one way but
that's only one at a time you really
kind of want to see a lot of output to
really gauge how your chat BOTS actually
doing like one outputs just not enough
so that's kind of like the first way
you're going to interact with your chat
bot in my opinion or at least see how
your chat bot is doing so I'm gonna open
up the nmt model directory and head into
the model directory so inside here is
where everything goes this is where all
your checkpoint files go and all that so
just in case I forget to mention it
later like let's say you want to go into
production and you want to make move
this chat bot somewhere else all you
need to run this model live is the nmt
code right you need to have you probably
want to bring h params if you don't have
the same exact settings file but you
could go either way actually but anyway
you're gonna want checkpoint probably H
params and then the three files that
correspond to what's which checkpoint
file you want so in this case this would
be step one thousands checkpoint so
you'd want all three and then you'd also
need to modify the checkpoint file to
contain the number right so if we
actually wanted a thousand we would need
to change this to a thousand okay
and you might as well change all of
these or you could just remove them
basically normally this would be like
496 495 494 and so on all the way down
but I just threw in 496 because I was
actually kind of happy with model 496 so
I just copy and pasted into all them
anyway the first form of output is gonna
come in these output dev so output dev
is the result of your chat bot every
5,000 steps is going to take your tests
2013 dot from file that's located in in
your data directory so here okay yeah
right there
so then oops I try to hit the back
button on my mouse and I'm in paper
space right now so it just actually did
my browser not here anyways as I was
saying so yeah so your first for my
outputs gonna be that output dev and
test but output David will look at real
quick but anyways so this is output from
our actual model so we can see some
examples really quickly of how our
models doing right now so that's pretty
cool and you can either manually compare
you know the the input and output or I
just wrote a simple kind of pairing
script real quick so I'm just gonna drag
this over and this is it so the output
file output dev the testing script blah
blah blah so give the full path to
wherever these are located then you can
run this and that's going to output line
by line and it's all just gonna look
like this so you'll you can more clearly
see hey here's the input here's the
output and this is really nothing
interesting here so I just don't see a
point for me to like write this out on
tutorial or video for you guys you can
go to the text-based version of this
tutorial it's all there so yeah so now
we're going to talk about the inference
script so I'm gonna close this out so
that's one way you know you can sort of
kind of interact at least see how your
chat bot is doing in Mass but then
there's gonna be a time where like say
you like the way that's looking you
might want to start interacting with
your chat bot and chances are you're
gonna come up with some questions that
you you tend to ask or you tend to see
oh these are problematic questions so
you want to ask them from time to time
to see how the chat BOTS doing
if it's the same questions you're gonna
ask to every chat bot though what you
should do is create is just add them to
your test your test file you'll have
needed to have done that before you did
prepare data and all that but just keep
that in mind like if if you're gonna
keep doing the same stuff you could just
throw it into the test and that way I'll
put dev every 5,000 steps is going to do
that for you but chances aren't you're
gonna come up with new stuff or you're
gonna see new ways that your chat bot is
kind of weak and then in each one's
gonna be a little different some chat
BOTS like to uncle over the place some
chat BOTS like to not finish their
thoughts some of my chat BOTS have liked
to give way too many links to things
that don't really need links stuff like
that each one has been a little
different so so yeah anyway the next way
is with inference so inference at PI and
things are highly likely to change so
I'm just gonna throw that caveat out
there right this moment if I go to the
github yeah this is what it looks like
and in here you have this is kind of the
way things are right now and then
there's send xlab and in there I've got
a few things that have changed including
a modded inference a modded bulk
inference and then some scoring
information now I'm gonna go over all
these in a moment but I just want to say
that as time goes on chances are the
scoring and the modifications to
inference and picking the right one and
again I'll explain that a little bit in
a moment that's probably gonna be
implemented in a much better way by
daniel later whoops yeah this guy anyway
i meant to just code here in the in the
main project directory so chances are if
it's enough time has gone on there's
probably a better way than using my code
so just I'm just gonna say for now
anyway
the default one is just inference type
PI that's the one that doesn't do any
scoring or anything so first let's go
ahead and run that one so I'm gonna go
ahead and open up a terminal change
directory into desktop nmt chatbot and
let's see if I can't make this big cool
so I'll run Python trained PI
and this is going to open up based on
the checkpoint director or the
checkpoint files so again if you wanted
to test a different checkpoint you could
go into model you could just edit this
checkpoint file that contain any
checkpoint you want and that will be the
model it loads
okay so just keep that in mind if you
want to check a different one and I
highly suggest you do because even
though let's say 496 and 4/9 let's say
let's say 496 thousand compared to four
ninety five thousand might be a
surprisingly different chatbot right it
might be significantly different it
might be a lot better and then suddenly
at 497 thousand it's bad again you're
not happy with it so I strongly suggest
you check different ones also it's not
always going to be the case that the the
one that was trained on more data is
necessarily better so for example I'm
comparing this chat bot this is one that
I created with 70 million pairs compared
to the other one currently running on
Twitter right now is trained and trained
on only about three million pairs and I
honestly think the one that's on Twitter
right now is better than this one so so
that just goes to show you that you know
it's not necessarily that the one that's
been trained more is gonna is gonna be
better so anyway and there's a lot of
other considerations to go into that but
just keep that in mind don't want to
test a few of them so yeah anyway coming
down here what have I done did I run
trained up high I must have ran trained
up pilot who practice I'm so used to
doing Python training up I'm positive
that's what I did let's scroll up and
see that was dumb I sure did what an
idiot anyway let's see if I it's not
gonna break until this operations done
so I'm just gonna start over open in
terminal make a bigger change directory
into desktop nmt python inference dot pi
anyway
cool so now it's gonna start the
inference and then it starts this
interactive mode first response we'll
take a walks it's gonna load everything
but hey there comma how are ya doing
okay so this one like I said this one's
gonna take a little longer and
the program says the other ones will
usually come a little quicker okay so
here you can see all of the outputs
there's quite a few here what's going on
well we're using beam search so that was
one of the parameters and the hyper
params and this is one of the benefits
of doing it is that you can get multiple
outputs now the default is 10 if you
want to change that you can just in the
model directory come to H params and
what you're gonna look for is both beam
width I think is what it was yeah beam
width which is probably set to 10 right
now I set it to 30 for the production
model and then it's like num
translations per input yeah yeah there
it is
30 again pretty sure the default would
be 10 but you can make that 30 so
anyways you can change those two things
and then you'll get up to 30 so now you
can see though hey there how you doing
you can see the first response was hey
okay which is a little worse than hey
how you doing right or how is it going
how was your day but really probably the
best one is to say like I'm good or I'm
good thanks right yeah and you can see a
few or you know I'm good thanks for
asking you know that kind of stuff and
obviously a lot of these hey hey hey hey
hey I like how this one just has a
random Hey back to the regular Hayes
anyway so so so this is a slightly
modified inference over the default one
that came with nmt so the default one
would just return like the Mook the
number one and your chat bot normally in
training you know like or if you were to
push it it would be the first return
whatever it happens to be now this one
has a few different scoring mechanisms
built-in one is for unk's there is no
UNK here so you don't get to see it but
anything that has an unknown token is
really not user friendly so we you know
you don't never want to return that
really honestly I think you'd never want
to return something that has an unknown
token in it that just looks really weird
but anyway you can see there's a lot of
them to choose from and the first one
isn't actually the best one we could
argue about which of these other ones is
the best but it's probably gonna be you
know one of these I'm good I'm good
thanks
hey how's your day I suppose could be an
okay one or whatever or just I'm good so
enter in my added layer on top of that
has been the modded oh yeah I guess I
don't really want to close this the
modded inference now I just wanted to
show this because what I did was I undid
this to go into actual production so
mines not gonna return all of these it's
only gonna return one response so try to
think there's anything else I want to
bring up here so basically what it's
gonna do is it's gonna read these
responses from an MT it's gonna use some
really rudimentary natural language
processing basically gonna look for
instances where we don't end on a
punctuation a proper punctuation like if
we end on a quote that's bad if we don't
have a period we're gonna we're gonna
say something with a period is far
superior also he had a propensity to
like not finish the links and the links
are like formatted like reddit links so
a lot of times he just wouldn't finish
and give that closing parenthesis so I
used a simple regular expression to look
for brackets followed by a opening
parenthesis but never a closed one
that's always gonna be a problem so
anyway so I'm looking for stuff like
that and we could I guess I could pull
up the scoring mechanism actually so in
syntaxes lab if you do want to use this
stuff basically you'll take all the
stuff you'll take comparisons modded
these two modded files and scoring and
you just you drag them into the root
directory if you want to use them again
though as time goes on I'm very
confident that that Daniel would
probably gonna write something better
than what I've written but we'll see
uh-huh anyway so you can look at the
scoring though it's just a bunch of
functions that kind of score so bad
responses so Charles b2 like to link
very frequently to this this list of
burn centers in the United States and it
just got really annoying so we stopped
letting that happen and then also some
of the responses would continue at the
beginning of a link but didn't finish so
though calling that a bad response
anyway continue on messed up link this
is what I was talking about it doesn't
close a link this is just a quick
function to remove punctuation for
evaluation stuff ends in equals yeah
basically a lot of links are we're
ending in equals so if you like
youtube.com/ and then the V equals and
then it would just stop
well don't really want that so if the
thing ended in an equals we lower the
score UNK checker there's any unknown
token remove four from the score if the
answer so a lot of times he would just
repeat the question which we've been
sawing that output already if he does
that remove from the score ends in
punctuation good we add to score if he's
just echoing if he's very similar to the
input question we want to penalize ants
more stuff on answer echo
buh-buh-buh-buh-buh anyway eventually we
get to the point where we return score
and then we're gonna score basically all
the outputs and then go with the best
score if there are a bunch of them that
have like say this score the best score
is six okay let's say so let's say
there's five answers that have a score
of six we just randomly pick one okay so
that is scoring and then now we'll do my
modded inference so for example let me
just do Python modded inference and hey
there , how are you doing I think it was
that hopefully he doesn't respond hey
we'll see how's it going so
unfortunately responded with a
relatively similar question but
obviously how's it going I mean a lot of
people when you say what's up they say
what's a backer hey how you doing hey
hey you done like they pretty much just
respond they never actually say how
they're doing so that might not be
totally invalid all of them
Wow that's a lot of cats that's a lot of
cats - let's see if he responds to this
up so I can't do that okay oh that's
gonna be empty we're gonna hit an error
no if you don't have any input it's
gonna get pretty angry anyway
wow that's a lot of cats let's just try
it again see if it gives the exact same
input we've rude that's a lot of cat are
you better than version two on Twitter
he's not I'm not sure if I'm better than
two on Twitter anyway yeah so that is my
modded kind of where it picks one decent
answer but I'm sure if we were to kind
of compare these like for example like I
could break out of here and then let's
ask him how many cats does he own so
let's just go back to inference top pie
and let's say you know how many cats do
you
we've rude okay so as you can see
there's a lot of answers to this right
he just repeats the question is the
first one all of them how many do you
own I don't own a cat so I don't know
another question of how many he owns or
how many the questioner owns and then
here's one where he doesn't finish his
thought I don't think I've ever owned a
cat but right and then weird binary like
answers and so on so anyway as you can
see I'm sure even if Daniel doesn't come
up with better scoring mechanisms I'm
sure other people will come up with with
better scoring also you know each of the
scores is fairly arbitrary as far as the
number I'm subtracting or adding if
things are good this is really kind of
catered to make Charles to be to good I
imagine that if I go through with a v3
model I think that probably I'll have to
tweak it again to get the results I'm
after anyway I think that's all for now
there's just really a lot of trial and
error and research and development
that's going on at this stage for me
like I said I trained a model with 70
million pairs I did a full epoch on that
data and I just really wasn't all that
impressed I even went back a little
further to see if I could get things to
be a little better the second time
around and I just wasn't impressed so
and that was with a 1024 by 6 by
directional model with about 70 thousand
vocab if I recall right just didn't like
it so now I'm doing is going back to the
512 by 2 bi-directional which is what
Charles V 2 on Twitter is right now
which obviously is going to change in
time but now I'm doing a 512 by 2 with
500,000 vocab on a 24 gigabytes of vram
machine I shrank the batch size to 32
which right now we don't have the option
in the settings at least I don't think
so in fact we should just add it right
now because it should be there because
that's one way you can keep a larger
model but not you know run out of
theorems
okay so that is there it's just
commented out so someone else must add I
guess that's the last okay so he added
that in or someone did it anyways so the
default is 128 but a good idea would be
if you're running out of memory I'm
doing 32 and I'm just barely fitting it
into that 24 gigs of VRAM so anyway I'll
see how good that model is I have high
hopes but who knows but yeah a lot of
this is gonna be trial and error and
yeah a lot of people were suggesting hey
you should do like rule-based Bart blah
blah blah and I mean to an extent we are
I mean I am applying rules to the output
and pretty much immediately everyone is
I'm not going with the first choice and
then we're doing having various rules
being applied to it you could come up
with other things like for example if
you're hitting unknown tokens well
that'd be a really good time to use a
Markov chain right so you could figure
out okay what's the best word to throw
in here you could throw in a Markov
chain or something like that to find
just to fill any unk kind of spots so
yeah there's definitely a lot of things
that we can do that we can make going
forward but this is probably be the last
chatbot tutorial besides you know just
some filler updates as time goes on as
far as what I'm figuring out but for
example the last model took about a week
actually a little more than a week I
think it was about eight days to do even
one epoch so it's just these things are
gonna take a long time and now the 500k
model at a 32 batch size is stepping
about just as fast and so that's gonna
probably take a month to do an entire
clock so I doubt I'm gonna do an entire
epoch with that model but yeah these
things are just gonna it's gonna take a
long time so anyways I think that's all
for now if you have questions comments
concerns you're finding you've you've
done something and you made some cool
change that you want to share leave it
below you can also come join us on the
github make a pull request or even 4k if
you want like I said some people thought
they had some good ideas and my answer
is always gonna be the same for get or
make a make a pull request and yeah so
contribute anyways that's all for now
questions comments lean below otherwise
I'll see you in the video