Welcome everybody, you're watching Mr Fugu Data Science. Today, we will parse
some nasty json data using new techniques. But, first we need
to go through the twitter api to retrieve these data.
I was asked by a viewer to put two of my previous videos
together to show how i did it. But, today i'm using new techniques for solving
this problem. I'd like to say thank you to the new
subscribers and recent viewers greatly appreciate it. You're gonna need
to import a few things so let's get that out of the way. You need pandas,
collections, base64, request, json, and default dictionary. You will need to
have a standard twitter account as well as a developer account
established before you get this done: if you are interested in the
api portion. So let's get into looking at this real quick you have this
authentication page that has useful functions that you're going to
use. We're going to scroll down and look at
the basic layout. Here's our template of how we're going to set
this up. It looks like you're doing a post
call, and you're calling an oauth 2. You're
retrieving data from the api, you have an authorization, where it says
something basic. And you have this funky looking thing
which would be your key. Then you have this content type
which is standard for your application that you're using
and then you have this grant type; that we need to snag as well.
So you would put your secret key and your client keys both here.
I would suggest using a config file if possible. We need to do some string
formatting: so we can actually send these keys off. Where we were going to use our
client key, a client secret key, and then we need to
do some encoding and convert that into ascii. You will
call in base64, then you need base64 encode.
And you're just going to call in the key secret that you just created.
We will then have to decode this: so we're going to call
base64, and then you need b64 code key. And then we're going to say
decode into ascii. Nope encode.
If i could spell correctly today. And we need to look at something called an o
auth. It's basically just a protocol giving your application permission to
act on its behalf, and you're not sharing any of the personal information
such as your password. There's a few ways that you can actually grant this
type of permission: authorization, client credentials, and
passwords. And you could look at the distinction
between these. I left some documentation if you're interested in going further
into figuring this out. We go back and we look at the setup and
we would notice that we're setting up our api. We're calling in your
token with your authorization of 2 your oauth 2. Which is using the
public data versus the private data for oauth one.
And you're using a bearer token, all right. So we're
calling in this authorization and you have that
basic and this is where the key is getting stored.
And everything is the same as the example i showed
in here. Then you call this in. So then you send this in and you get a
response code of 200. If you make too many calls repeatedly in
a short amount of time you get an error: it's going to stop you from that.
Just be aware. And setting this up in basically the order that kind of
falls, all right. Then you have the json response which
is basically token type and your access token. And if
you printed it off you see it's just this jumble looking like hash
looking thing, if you're curious. We'll do the oauth one for a get request
and you have some parameters. We're gonna look at those parameters
and you have your only required one here which is basically giving you a max
character length 500 including your operators.
Im also going to use the count which will give
a max of a hundred items per page i'm only going to do 50.
You have other optional parameters that you can use. We're going to
do our get request we're using our oauth of 1.1 calling in that api.
We're going to have the tweets in json formatting.
And this q is your search. You have three options here you have
popular, recent, and mixed. You have in this case
they're looking for results of Nasa and then it's all the
same other stuff if you read into this. Now,
we're going to look for climate change today, and the most recent
and it's basically the same setup as the post.
And what does the actual tweet look like when you're doing this? The tweet looks
like this so you have an outer portion which would
be your statuses; and then everything falls in as a list of dictionaries of
which some are extremely nested. Using the text
which is the actual tweet is what most people will use
for some kind of natural language processing. But, in this case we're going
to delve into it and practice parsing.
That's the data. So if you only wanted the tweets you would stop here and you'd
iterate through the statuses and look for the
text. Which is inside of it, and i decided to print off the first text
here. For our convenience we're going to take the statuses and keep everything.
So you can see that we have the dictionaries, and how long is this
it should be 50. We store this into a list. Then let's look at what this
would be as a data frame, and that's fine. It looks like everything's good; until
you scroll over and you see here that you
have dictionaries with lists for values. You
have other dictionaries. You could keep scrolling on. Now this is
what we're going to play with there's a few ways you can go about this
first thing is looking at the columns and you can actually pull all the
columns that are objects: start iterating through those and figure out which ones
are dictionaries or not and manipulate from
there. If that's what you wanted to do. That's one route. You can look at this
by row wise and consider looking at all of the columns
and see which ones print off dictionaries and then manipulate from
there. I decided that i'm going to use the data
frame route. So let's look into this. Let's take our data and create just some
dummy random list. And i want to iterate through this
for the json tweet data and let's take our statuses. and we need to then
iterate within side of the data. Let's go inside the statuses.
We're going to iterate through each one of these. I need to convert it into
values. And then if the type of j
is an actual dictionary: then i would like to append i and j. Where
i is basically the index value. And that's what's going on here. So
what's up with this? Let's look at this a little bit. I'm going to take the
first entry of this that i converted into
values. So it's taking this and converting it
into a list and this is the first entry within that list.
And what i'm looking for specifically is anything that's a dictionary.
Any of the other values that aren't a dictionary: i could just
call those out later and concatenate it to the data frame. But, you'll notice that
that's not needed or unnecessary. Because, this same data right here is
stored within the nested data we'll find later, all right. So we have
now basically the row number that all of data is corresponding to as our index
with all of our nested entries. What do we do with this? We
need to create a default dictionary list
so we can store our key value pairs, okay. Where our values
are going to be the list. So let's iterate through this and we're going to
append to our dictionary list: the key which is
our index and we're going to append our values
into this. What's this look like? Here's our first entry where we have
a list of all our nested dictionaries, okay so we're
getting somewhere. Then we're going to take these data and
just name is something fun. Let's just call it df yay. So i'm going to
call in a data frame: so i can store these dat.a
But, i need to do something really interesting here,
okay. I'm going to call in my data but, i'm going to take my key
and convert my value to a series and my key will basically be
the column number and then we need to iterate through k
and v in the list that we just made. We're going to call this as a dictionary.
So it's going to get the items and let's see what this looks like, okay
this is interesting right. But, it says that we have 50 columns when we should
have 50 rows. So we need to transpose this.
Now we have four columns and 50 rows which is
more like what we need, But, what's going on with this? Why do we have four columns?
Well let's split these up into distinguished
dictionaries, which is very useful. And if you notice at the very beginning you
always have something that's created at, all right. We're getting somewhere
pretty useful. How can we deal with this now because
it's all dictionaries of dictionaries the values of list
and miscellaneous information. So let's look into this. Let's do some
converting that's where the import json is going to come
in handy. Let's make this some variable
i don't know json structure i guess or structure however you want to think of
this. And it's called json loads which is for
json strings. And we're going to
convert the data that we have into a json object. And we need to orient this
as records. And what does this even look like? This is kind of funky right.
Okay, so we got a list of dictionaries corresponding to some key which is our
index. Okay, so let's see how this actually gets used.
We can take our, we can take our data and we can say
json normalize we take our little
struct that we just created. What does that do? Ooh that's interesting.
Let's look at something real quick. You have your data which is nested
dictionaries and you now unnested it to a point.
I say this because, these values here were actually list of further
dictionaries; so you need to do further processing.
We started from however many columns here.
You started from four columns here your nested data now exploded into 121
columns. What we need to do further processing on
this right here.
Okay, one thing you can look into is saying there's an option that can
work in separate circumstances i've never covered before which is called
explode. Which would be calling in our data frame and then you call in the
explode and what column would we like to take, let's just make one up.
Let's just take this one and see what it looks like.
So if you see here: there is a difference because, it got rid of
the list. I made a function that i'm not using
today which was shown in my previous video
and i'll put a card for that where you can actually separate this column here
after you've got rid of the list and unnest that.
This is one option depending on what are your circumstances. That you can use. It's
pretty useful: except in this case it partially helps. What do we do from this?
Let's break up that column with a list of dictionaries and see how we can do
this. Call in our trusty old
i don't know let's call in this column, and see how this can work.
Just throw in this specific column and let's use an apply function and i'm
going to convert this into a series. So i'm taking this nested data
and i'm going to convert it into a new panda series.
We're going to use the json dot loads once again. Like we did above, take in
our nn variable. Convert this into a json, and we're going to orient this equals
records. Like we did in the previous example. And then we're
going to do the json normalize for the
new variable we have and let's look at what this turns out to be.
Okay, so what's going on with this? Looks crazy, you have a lot
of columns now. So what happened we took in
this specific column and we created more columns.
All right, so we just basically broke up that nested data
into these columns. It turns out to be 18 columns if i'm correct
from that one column that we just manipulated and mind you you have 40 or
so more columns to do the same type of
thing with. So this is going to expand to way over you're going to be at more
than 238 columns that you have to sift through.
Needless to say, let's look at this. So in doing this i generated columns that
were duplicates and i want to remove those. But this
first concatenate onto the data that we were dealing with
and i'm going to take in this.
And so now we have 135 columns in this circumstance. Each time you run the
twitter api you're getting new tweets sometimes when i did this it was 238
columns at this point it's 135 columns. But, let's find out how much of those 135
columns pardon me are actually duplicates. So
we're going to drop some columns. And i want my axis to be along the
columns. But you need to say how do you want this to happen.
Usually it just says any. But, i want all. So if the column has
all n a's drop it.
Okay, so we have 115 columns now instead of 135.
Why do i have 100 rows i should have only
50 rows. There i am, that's what happens if you're not paying
attention. We want this along the column so we keep
the rows perfect. Let's look at something real quick. Let's
print this off. It looks like in this circumstance of
running these data i had 14 of my columns which were duplicate
columns. Which was 1 less than what i had before, which is
interesting and you have to take that into consideration when you're using
these data. Now i didn't move this
along further because you have to run this basically for each one of these
issues you have for any one of these columns. And so
in order to finish this up and take care of each individual column and explode
this into a really wide data set you'd have
to iterate through each one of these column wise and do a comparison
of basically either data type or use a combination of functions. But,
that's up to the end user to figure out how to do it. I just showed that you can
take that one column here convert it into multiple columns and unnest it.
That's what i showed here: i showed how you can use the json
normalize to flatten out your data. And i also showed the explode: how you
can get rid of list if you had a column or your values
were a list of multiple values. How you can use explode to get rid of that list
and if you had like a tuple or if you had something like that to be able to
line up for the rows. Another, thing i would suggest is
changing the column names and getting rid of this because
that notations for what the original column was that it
was nested within. You could do some string formatting for these names.
Anyway i would like to say thank you for watching. Please like, share and subscribe
and if you subscribe turn on that notification bell. I hope this brought
utility to someone. See you in the next video.
Bye