hello everyone welcome to the semicolon in
this tutorial we are going to learn how to make
chat bot an application which is going
to talk to us, based on the data which we
train it on i am very excited about
it and let's get started in this
tutorial we're going to learn how we can
model the data or text data to feed it
as an input to our LSTM neural
network so let's get started so I have
to import all these libraries and I am
changing the directory it seems to be a
bit of problem and I've got errors
already okay, I have to execute this and
the errors or because of the slash here
which I have now removed and I am importing
a pre-trained word2vec model from
the AP news corpus which is available
for downloads and i will mention the
sources in the links below, my data is here
in the conversation.json file
which I am loading which is in the
corpus folder in my directory of the chatbot (directory)
and doing this so what I have here
is all the data, you can see our
training data looks like this good
morning how are you I am doing well how
about you and so on so this has many
such data and we will be training on a
short data set of 86 86 sentences but
you will be shocked to see how good our
model is based on this small data so now
what we have to do is we have to model
the input and output based on this
corpus so what we'll do is we'll have
questions and answers
so questions will have this one sentence
good morning how are you answer will
have I am doing well how about you
then question will have I am doing well
how would you answer will have I am also good so
so then we can generate 86 question
and answers like this, we create two
lists x and y and then we try to
generate our model then, what we do is
we are trying to get our data and append
questions in X from this from this
variable so let's start doing that and
we will have our data in X and Y soon
the X and we have not executed this and
now we have questions and we have
answers and then what we do is we
tokenize it and tokenization is
separating each word so now we have a
sentence to be tokens of words so it is
good morning how are you so each one is
a token now what we do is we create a
vector of all ones so that we can append
this at the end of a sentence and now
for every word in our vocabulary or
every word in our corpus we are trying
to find a vector for it in our word2vec
model and once we find the vector
we creating a sentence to be a vector of
word vectors so our sentvec would be
the vectors of word vector which we are
appending in this list and let's do this
and this is done, so similarly we do it
for Vec_y and we get all the models and
I hope you can see the variables here so
our variables are visible here which if
we load are too huge which may crash
the
IDE so now what I'm doing is I'm
clipping off all the sentence all the
words which I have after the length of
15 so after the length of 14 I have is
null array I am removing I am deleting all
the words here so let's do this and make
all the sizes same now let's wait for it
I will show you what the dimension looks
like and now I'm appending sent end to
make the size of the word to make the
size of sentence 15 so if a sentence has
a word hi then it will look like hi and
then 14 vectors so that is how our
model our data is going to look like and
I am doing the same thing for y as well
and you will see why it is important and
then I am appending all these sentences
with sentend and I am saving this as a
pickle dump so this is important because
in our next tutorial we can directly
instead of this pre-processing of our
model we can directly load the model and
then use it so that is what we are going
to do in the next tutorial so we have
our dump as list of vec_x and vec_y which
are our corpus in the gensim word2vec form
have to remove this and i'll be
uploading all the codes on github so
you can get it from there and implement
your own chat bot so hit the like button
hit the subscribe button and keep
watching thank you we'll continue this
in the next tutorial