what's going on everybody and welcome to
a video where we're gonna be going
through some of our Google takeout data
if you're not familiar with Google
takeout it is a way you can go to Google
like your Google profile and download
all of the data that Google has on you
at least through Google services so if
you want to do that it can take a while
to actually get the archive but if you
log in you can go there's a blue you
know account top right click on that and
then on the left hand side there's data
and personalization you can click on
that and then from there scroll down
just a little bit and then you can
download your data and then you're gonna
pick the things that you want now you
might go through this list and you might
think you know what those things mean
but you probably don't so just get it
all so you get a good idea of all the
things that Google has on you because
you would actually be kind of surprised
or at least I was like there were some
things that I thought I understood like
I thought I knew like what kind of data
Google had on me but I didn't and so
it's just a good idea just grab it all
just so you have a full idea of all the
information that they have so it's even
like little stuff like they've got like
Purchase History for example even though
you might not shop on Google but Google
parses your emails and from parsing your
emails they've got Purchase History from
like confirmation emails and hotel
booking emails all that kind of stuff
they're just extracting that by parsing
your emails so stuff like that you you
might not realize they have so anyway
definitely I think just get it all but
mainly we're gonna be focusing on the my
activity section at least to start but
later on if you guys want to suggest
stuff that you want to see us go over
feel free so we might do other stuff so
finally you'll create your archive and
this will take like at least for me it
took a few hours and you'll get like a
confirmation email and stuff like that
like for security I can't think of a
much more scary email to receive them
you've requested your Google takeout
data and if you weren't expecting that
email so anyway cool so then say you get
the data and then once you get it you'll
get like a ticket like you'll extract it
and you'll have a takeout directory kind
of like this this is not the full one
for me
but just some of the stuff I wanted to
point out so this is that purchases and
reservations thing I was talking about
location history this is you tracked
everywhere you go based on your phone
but not just everywhere you go it's not
just coordinates it's are you walking
are you staying still are you in a car I
didn't think they were doing that but
they are and based on that you can you
can then extrapolate information so
based on that you know where the person
is like where are they sleeping at night
basically that's probably their home
where do they go every day Monday
through Friday
same similar times that's probably their
work you know or school or something
like that and then in based on the
coordinates they can pretty easily find
out like what's at those coordinates and
stuff like that so anyway like I
remember the first one of the times I
moved I went to the same store like
twice in a row or something and Google
just assumed that's where I worked it
was really weird but my phone was like
you need to go to work soon one day and
I was like what are you talking about I
have a job so anyway yeah so we're
interested in my activity inside of here
there's stuff like Android again one of
the things I didn't think they'd tracked
was like every app you open look they
just track it all right cool and as time
goes on like all this stuff like on its
own or alone doesn't really sound that
nefarious but like then you start
realizing like all this stuff adds
together to basically be your whole life
and it's kind of creepy so anyway on
that note we're going to be going
through our entire search history and
hopefully you know first of all just as
a forewarning you guys don't get to
comment on my search history unless you
post your search history so no making
fun of me so anyway this is what it
looks like it's an HTML file which is
kind of weird because other stuff like
for example the location data let's just
pull up that that's a straight-up JSON
file so I'm not really sure why this one
is an HTML file and like some stuff is
JSON and stuff I don't really know but
anyway so we want we actually in this
case we're gonna have to parse this HTML
file which is kind of a pain in the butt
versus like the location history like
getting through that is like a breeze
you just import JSON and you go and in
this case you could use beautifulsoup or
something but we're
just gonna do some stupid splits I think
I think that's the way to go so the
first thing we're gonna do is just
something really really basic we're just
gonna run through all our search queries
split by word and then just start
looking at what are the most frequent
words and then what we can do is you
know oh you could do that overall but
this is like five years of search
history so then you could do like a
daily moving one year window so like
every day along the way what was the
previous year's worth of search history
and what are the most common top ten
words for example and then and that
should probably give us like a good
overall like macro understanding of
major interests of ours and major life
changes and stuff like that and then we
could go smaller like month a month
window or a week window and figure out
you know micro things that were going on
just at that time so anyways pretty cool
so that's the plan later we could do way
more advanced stuff like do like word
vectors and get general concepts that
you know I'm interested in or you are or
whatever we can do lots of really cool
stuff one of the other things I saw that
I absolutely have to try is they also
have all of your google assistant data
so you're the actual translation is
mapped to the audio which is basically a
text-to-speech data set waiting to
happen so and also a speech-to-text
mm-hmm so pretty cool but mainly so you
can create a text-to-speech but this
sounds like you also take note Google
could do the same great so anyway let's
jump in so first of all I'm just going
to make a new a new folder I'm gonna
call this G data I'm gonna put takeout
into G data and then I am going to
figure out why my mouse isn't showing up
hopefully oh I can't I just can't see my
mouse okay cool
alright and then what I'm gonna do is
file I'm gonna save and I am gonna put
that in desktop G data and then for now
I'm just gonna call this on a search
database dot
hi search cool awesome so that's all
we're gonna do we're just going to parse
through this save the times we're gonna
convert those times to unix time because
the date stamp is going to be kind of a
pain to work with we're gonna split by
word and then save that into our
database and scratch all of our itches
okay let's get started so first of all
we're going to need import import
sqlite3 for the database we're going to
front well actually probably in this
file we don't need the collections I was
going to bring that in let's go ahead
and bring we're going to use from TQM in
40 QT q DM s QT and then we're gonna do
import date util dot parser and we're
not gonna need JSON I'm gonna still
scratch it is I'm sorry guys I got like
allergies or something I don't know well
we'll just get started so first of all
let's get the search activity at
activity location so I'll just do
takeout my activity search take that
copy pasta slash my activity dot HTML
don't know if these back slashes will
actually create a problem in this case
but I'm gonna just fix those manually
real quick then what we're gonna after
this we're gonna create our table or
actually first we need to make the
database which is actually super simple
with SQLite you just connect and if it
doesn't exist boom it's created so let's
do that so we're gonna say con equals
SQLite 3 dot connect and we're gonna
save this as my life dot database and
then C equals conduct cursor awesome
define make table now what we're gonna
do is C dot execute and we're gonna
create table if not exists the table
will be called words and basically words
is gonna contain
I think we'll still have a primary key
why not
id integer and that will be a primary
key ID we're gonna be UNIX that will be
a real we could also go with integer but
I'm gonna say real and then the actual
word itself which will be a text type
awesome okay so that's our table can i
what's what's right here indentation can
eight I'm so glad that this thing is
like that's that's violating pepp eight
no tabs allowed let's fix that really
quick indent using spaces fix this
stupid idiot awesome okay so make table
so let's go ahead and we'll run make
table we don't actually need to run that
now though and now what we want to do is
work on search data so define search
data what we want to do here the first
thing is we're gonna split by word but
we don't want all the words like a V and
these are words we don't care about
those and the NLP community are called
stop words these are just words we don't
we just want to kind of toss them out so
the first thing I'm going to try is stop
word list and list NL TK's list of
English stop words heck yeah that is not
the format I wanted but uh here we go
here's a list anyone want the above list
as an array here you do yeah thank you
because the other guy obviously isn't a
programmer Union this looks like a great
list I'm taking it
boom
copy nice big long list so we're gonna
say stop words equals bang it's kind of
a this is not the greatest list of stop
words Thoreau has meaning wonder also
has meaning seriously has meaning I
think I'll use this list but it depends
on your depends on what you're trying to
do I mean some of these are actually
meaningful words I'm not sure you'd want
to toss them out in our case I think it
will be fine
but it's interesting some of these I'm
just not sure I agree but the show must
go on so now we want to do is we want to
open oh we've already closed it but we
want to open that HTML file we want to
split for searched for I gotta open it
cuz I gotta figure oh wait is this it
nice we got a split for because we don't
care about this we care about searched
for so we want to split for that and
then we want to parse out each word in
that link so pretty easy task so search
data stores so now we want to do is open
that file so what we're gonna say is
with open the search activity with open
search activity with the intention to
read as f contents will be equal to F
dot read then we want to split those
contents and iterate over those contents
so we're gonna say is for item in
contents dot split and we want to split
by searched for with a capital S okay
cool well let's just print item print
item and then we'll break because we're
going to do some development here we
don't want to well we made the table but
we probably didn't do search data we did
not search data let's try one more time
error car map blah blah blah blah blah
okay fine we'll open it with some
encoding wish this was somehow he
here to get back over here here encoding
equals and we'll make that UTF Adel's
and we'll see what happens
beautiful not that beautiful but it's
what we wanted oh so when we split by
search for actually the the zeroeth
element we don't actually care for so
let's do 1 : try one more time
beautiful exactly what we were hoping
for so now we're gonna do some pretty
ugly splitting here so I'm gonna say
search underscore string equals item dot
split by that back escape that there and
then it'll be the first if dot split by
the closing link tag here split by boom
and in that case we want the 0 with so
now let me print search string beautiful
just beautiful
now we want to parse out the date where
is date here so it should be the first
break someone's gonna be like this is so
ugly in the comments whatever bro date
equals item dot splits break you did
this so stupid look you could have done
it this way okay split div and then 0 if
here's a ring X string this would works
so much better okay I'm done for it and
then date and let's just stop printing
item let's just see how that looks
beautiful beautiful now we want to
convert that date to a unix time so we
can use it as the program as we are so
date util dot parser dot parse the date
dot timestamp
that's not gonna do anything for us d
equals date rather than date we'll just
print D print the D looks good
TZ name EDT identified but not
understood past easy infos argument what
if I don't want it like I don't really
care about EDT wonder how do i what
steezy infos can I just
TZ infos date you to know what the heck
that's about TZ infos TZ and Posey
additional time stone names okay because
EDT I don't even know is that the same
that's probably the same as est like
Eastern I don't know what the D stands
for it Eastern date/time I don't know so
this is probably the Delta from what do
you call it u CT I don't know uh I'm
just not worried about it right now this
is working so I'm just gonna skip it
like a real programmer yeah the other
thing you could just parse up to like
the last four or you could past easy
infos or replace like what if I did this
replace EDT with est what happens in it
okay you don't even understand est okay
well that I don't care to cater to you
broke okay so we've got the search
string we've got the time now we can
insert the words into a database so now
what we're gonna say is for for W in
search string dot and we're gonna split
by a space
cool if W not in stop words what do we
want to do we want to see dot execute
execute insert in to insert into words
oh man I'm gonna flop here I I think
it's like this words I'm gonna try this
UNIX word fudge I can't remember if it's
like this and then it's like - I'm right
I'm gonna check don't worry that that
and then I want to say it'll be a comma
tuple and then W no no no UNIX is first
whoops oh gosh um this is going so fast
so DW and now I'm beginning to wonder if
maybe you don't have a comma there
that's looking good what do you guys
think I think I made it let's see let me
check this is the best programming
research resource I've never heard of it
it's phenomenal
dynamically inserting into a database
dang who is that guy
he's good-looking I didn't do values
dang it I was close I was close I
deserved partial credit values so like
that and then question mark question
mark comma tuple is that about right
close comma - yep
I was so close everybody okay cool break
I think that's it y'all so let me get
rid of that break and before we forget
you need two things we need a conduct
commit and we needed conduct close at
the very end now this is not too much
data like our full data set is 80
megabytes so there's no way this is
gonna be bigger than 80
probably to be like less than 10 so
committing at the very end is totally
fine many times you might not want to
commit once at the very end like you
probably wanna commit in batches but in
this case the entirety is smaller than a
batch that I would commit as so we'll
just commit at the very end no big deal
we're gonna comment that out and we are
gonna TQ DM this operation here so we
see where we are percentage-wise along
the way and finally we are going to try
and accept because we are probably going
to hit an error somewhere and I don't
want this whole thing to flop because of
one stupid error so we're gonna say try
cuz like some of this stuff you've got
like visited and then searched for so
that's kind of awkward and that's
probably gonna cause an error and
there's probably other things too so
we're gonna try except exception as a
print what what no no no what's going on
what is going on oh now you want to fix
okay whatever print string II look it
this thing is just throwing a fit what
is your problem lint her are we good I
think we're good okay so let's run this
GData
CMD looks like we actually practice we
already ran it so let me just delete
that and let's just be a problem but I
forget why we had to rerun anyway Python
search database ha we can do it here we
go we got a progress bar it's looking
great or how we're going quick just for
the record if you want to look at your
database like I'm going to and this is
done I can't really see anything yet
because we haven't committed anything
but I'm using like it's like SQLite
database viewer or something open with
DB browser for SQLite just for the
record we're gonna wait for this oh it's
done
three point four megabytes that's really
small okay browse data this could be
risky everybody I'm about to show the
world I could always blur it and editing
I guess but then I'd have to edit
okay so cool looks like it works we got
the unique ID a UNIX timestamp and then
the actual words I'm not going to scroll
too far that can be dangerous all right
so we are ready for the next tutorial
which is where we will begin to build
the images that will represent the
frames in our video of our search
histories over time that's gonna be
pretty cool
shout out to my most recent channel
members Rodrigo claw NYU Ian Kristensen
and Khun al thank you guys very much
without support from people like you I
wouldn't be able to play with Google
data like this I mean it's pretty
awesome so thank you guys very much for
your support and I will see the rest of
you guys I love you guys do in another
video