hello everybody and welcome to part 6 of
the chatbox tutorial series with Python
in tensorflow in the last tutorial we
built our database we inserted all the
roads and at this point I'm assuming you
guys have created a relatively large
database of pairs and then in this
tutorial what I'm going to be showing
you guys is how you can create training
data from that database when it's done
if you don't have if you have less than
like a hundred thousand pairs I wouldn't
suggest that you continue following
along unless you're just kind of curious
to see how things work so with that
let's go ahead and get started so what
we want to do here is basically to
create the actual training data that
we're going to use for our models we're
gonna be using we're going to kind of
poke around with a few models in this
series but pretty much it's always gonna
be the same kind of format and the idea
is that generally what you're gonna have
is a a from file to a to file so again
this is all kind of this basically I
don't even sure if this is again but but
basically what we're doing is tensorflow
sequence to sequence ok so whether it's
a chatbot which is a comment in a reply
or it's a language translation which is
what a lot of the sequence sequence
tutorials are doing or it's could be
anything I mean everything in life
pretty much boils down to a sequence to
a sequence it's not really so much a
fixed input to a fixed output it's
variable length input variable length
output and that's what's really
intrigued me about sequence to sequence
and especially some of the later
implementations of sequence to sequence
from tensorflow that's what's pretty
exciting about it anyways back to planet
Earth what we want to do is create
basically a parent comment file and then
a and then a reply file where each row
or each line number corresponds to the
other file okay so line 15 and the
parent would be the initial comment and
then line 15 in the so line 15 in the
from file is the parent comment and then
line 15 in the to file is
the child the reply to that parent
comment okay so in order to do this
we're going to import sqlite3 we're
gonna import pandas as PD if you don't
have pandas installed pip install pandas
then we're gonna have time frames and
basically I built this with it in mind
that I might have many different
databases with different times 2015 five
but really I think that you know you
might you might you'd probably combine
them most likely if that's what you're
gonna do but anyways I'll just leave it
that way and then we're gonna save for
time frame in time frames what do we
want to do so what we're gonna do is
we're gonna build this connection time
time frames we're gonna do is build this
connection and then use read SQL from
pandas to read it no you actually don't
need to use pandas to do this I'm just
gonna use pandas because there might be
times when I want to add a little bit
more functionality a little bit more
logic to to the to the SQL kind of pull
here and or even just data manipulation
or whatever and for that reason I'm
using pandas here but for what we're
gonna do in this tutorial series at
least for what I know I have planned out
I guess you wouldn't mean to use pandas
but I'm gonna use pandas
so anyway connection will be sqlite3
contact and we will connect to that
database so DB I did it again DB formats
time frame see the cursor is equal to
connection dot curse or and then what
we're gonna say here is first let's have
a limit equals five thousand we'll say
last UNIX equals zero cur length equals
the limit counter equals zero and test
done equals false you should know what
all that means so limit will be how much
we're gonna pull at a time to throw into
our pandas dataframe last UNIX will help
us to basically buffer through our
database so we'll pull we'll grab the
last UNIX timestamp of that pull and
then we'll and then from there we know
okay
in our next poll that say eunuchs must
be greater than last eunuchs and so we
just keep doing that with in each poll
has a limit of whatever this number is
in this case it's 5000 eventually we
could raise that mostly I want 5,000
because test is not done yet
so generally yeah you're gonna have a
two or a from and then a two file but
you also want to have testing files
something out-of-sample just to see how
the model is doing so we're gonna use a
test file and that test file will be the
first 5,000 rows of data you can make
this anything you can do 500 you give
you 50,000 you can do all hundred you do
whatever you want I'm gonna say 5,000
for now but yeah you can you can do
something else if you want so then we're
gonna go ahead and do is we're gonna ask
the question while cur length is equal
to basically whatever the limit is that
means we were able to make a poll that
that completely exhausted whatever a
limit was so chances are either there's
zero rows left but we'll find that out
in a moment or they're still rows left
so as long as we're able to get our
limits worth from the database we
probably have more polls to make so
we'll keep making polls so then what
we're gonna say is DF for data frame
equals panda so PD read SQL and what
we're gonna read books read SQL what
we're gonna say is the SQL statement so
we're gonna select for now we'll just do
all from parent and reply where UNIX is
greater than something and and this
should be all caps and the parent not
null and school is greater than zero but
it sure as heck better be order by unix
ascending limits
something okay dot format and basically
what we need to do is UNIX needs to be
greater than last UNIX so it starts at 0
and then limit now I guess that's the
only form we just did UNIX and then the
limit yes that's all the things that we
formatted awesome so that's it format
that and then finally the other thing
when you do a PD re SQL you pass first
the SQL statement and then you pass the
connection so connection boom
now come down here we're gonna say last
underscore UNIX equals DF Tale one so
the last thing unix unix dot values the
0 with boom so now we've updated that
last unix cur length let's see what's
the length of the data frame it should
be whatever the limit is now we're gonna
ask if not test done we're gonna width
open we'll just call this test up from
with the attention to append and we're
going to specify the encoding as utf-8
as f what we want to do is for content
in DF parent dot values what do we want
to do we want to F dot write content
plus a new line something felt wrong
content flows new line okay and then we
basically want to do the exact same
thing with test2 so with open test two
and then this should be comment so those
will match now if the test
oh well then also when we're done we
better say test done it was true also
after this point for really like
probably right here what we would do is
if you wanted you could update the limit
so we've already done the limit check so
we're good to go
so you could in theory update limit at
this point mmm no that would get angry
you'd have to update cur length and
limit temporarily if you wanted to do
that I'm not gonna do that but anyway if
you wanted to now would be the time next
what we're gonna say is else
so assuming test is done we basically
knew the same thing so I'm going to copy
this paste and then we're gonna call
this train and train again this is
probably best as like some sort of
function where the only parameter is the
name of the file so like test or train
I'm gonna pass on that right now but
yeah we could improve the script by
doing that now what we're gonna do is
basically wow that cur length equals
limit let's go ahead and counter plus
equals one and then if counter modulo 20
equals zero let's print let's print
counter times limit rose completed so
far so in this case counter modulo 20 so
basically is gonna be like every 20
times the counter so you'll see this out
so in this case I'm sorry every 20 times
the limit you'll see this printed out so
it would be five thousand times 20 so
100 thousand every hundred thousand rows
completed we're gonna get this
information so let's go ahead and save
that I'm gonna run it just to see if it
works
but I I didn't I don't have a full Pole
so I'm just gonna stop this whenever
it's done
maybe we should have chosen a smaller
number okay never mind
okay so we completed let me pause this
and we check those files make sure they
are correct okay so here we have our
files here's our testing so test from
test two let's go and open that
so tests from two so aren't they
streaming it for free online yes yes
they are that poor bastard so I don't
know I guess he bought bought something
so we'll continue down here so basically
you should have only five thousand rows
so I mean like the rows need to be
exactly the same same thing though with
train from into we should be able to
open those up and again like line 28
corresponds to line 28 interesting
thank goodness for utf-8 right it's
funny dogs give the butt of approval I
just keep finding just really golden
lines here here's our new line character
okay great so that's what we need to do
to get our data if you did a full poll
obviously you're gonna have much much
much more data in our case we just have
this data here but hopefully you can
have a much much larger data set than
just this such a short a little bit of
data alright so that's the end of this
tutorial
in the next tutorial we're gonna
actually start talking about the the
models that we're gonna use there's at
least two models that we're gonna be
talking about so it's what you guys have
to look forward to if you have any
questions comments concerns whatever
feel free to leave them below otherwise
I will see you in the next tutorial