Welcome everybody. You're watching Mr. Fugu Data Science. As
always the code will be in the link and description below. Thank you, to those who
are watching and who subscribed. We are converting JSON objects to data frames
using four examples. Then we create a function allowing us to store new nested
and non nested entries. Finally, discuss briefly how large datasets can affect
the timing of operations regarding optimization showing an example. Please, import JSON, pandas,
time, numpy. Okay do PD dot read underscore JSON. Call your file name here:
it's employee data dot JSON. Print it and see what your data looks like. Looks kind
of funny, looks strange. We have a features column with
rows of nested dictionaries. The original data had a key of features: where, all of
the values are a list of nested dictionaries. Okay,
that makes sense then. Let's use another method with open; call your file name and
then alias that has data file. And we need to iterate line by line in your
data file. Store variable name data call json dot loads, your lines of data, create
a new variable where you're creating your data frame of your data. See what your
data now looks like. Ah it looks the same, hmm. Well, even
though I showed these two examples they're important. Because, if you time
them, there's timing discrepancies. One is faster than the other. I'll show you an
end how you can look at that. Now, let's use our for ways to create our data
frames. We're going to take a read JSON, okay. So create a variable named DF
where we do our PD dot read JSON again. And take your file data, and then you
want to say: create a new variable with whatever name you want to give it up
make sense. Create your data frame of your data. But, you're taking the features
key, taking all of the values, converting it to a list, then you call the candidate
to go inside of your nested values now. So let's piece this together
okay. So you're taking your features and all of the
values here. Which gives you the list of the the dictionaries. Then you convert it
to a list, here. Which is separating all your candidates out. Then you're going
inside and you're calling your candidates. Which will go in here and just print off
all of these, okay.
Now we got that, you need to create a new data frame of that. But, that's not
going to work. Because, I that gets you here. so what do you do? You do: from records. And
then that'll get you going. That's a first example, perfect. Let's do
the same thing with JSON normalize. Take the first two portions, paste those
in. Create a new line that says PD dot JSON underscore normalize. Call
the BN. And there we go. This is a little different because it's taking somewhat
structured data and and flattening it into a list. But, it's very useful we can
iterate the traditional way where, we say for I in your data. and you call your
features. Where data was created when you did with open, okay.
But, we need to also create some variable to store our list so then we can now
append our iterator, where we're going inside of it and calling your candidate.n
and then we create the data frame, of our list. Let's see what that looks like.
There we go now, we got our third example. Okay, list comprehension which is great.
Just name it whatever makes sense to you. Now, we have to say let's take a
value for some sub list in our data. Where we're going inside of our features
key, iterate once more inside of that same
value in the sub list dot values, okay. And then create your data frame; and
there you go. then I created our fake data. Which I won't go into here; but,
here's what it is. We're taking hiring dates salary and health care which we're
going to add to our existing data. So that comes up here. These are the new
columns added to our data. We need to create our function which allows us to
take this dat a frame convert it into a JSON file with nested objects or un-nested objects. Were gonna do that now.
So lets define our function. as DF underscore to nested JSON. Where we're taking in our parameter
of our data frame, the candidate, the time somebody was hired which will equal
their hire date, the salary of this person, and the do they have healthcare.
Let's make this look better, okay. From there create a variable; which will take
our features as dictionaries. Oops sorry, a list of values. Then iterate through
key value pairs in our data frame. Using DF dot
iterate rows (iterrows). And we say create a feature variable taking our candidate which a
store dictionary values. And we're creating an
HR related section. Which will take dictionaries and this one this
example will do nested entry creation. Okay, so we need to do: from our nested
nested entries. We're creating lists of entries. We need to now do
a feature. Where we're saying; let's take our feature, candidate.
And now we're saying: we're going to put inside of our related
HR portion. The hiring information. So when somebody got hired and you're
calling that from the rows: where we're taking our values and you're calling the
hire. was this? No it goes like this. There cut and
paste for the salary and healthcare. Do the same thing;
salary, healthcare. Okay, now you need to do iterate and just say I don't
know prop in candidate. Then call feature, you're a candidate, and then what
you're putting inside of it. And you're putting inside the values for this.
Then you need your underscore JSON. So you can take it and throw it into the
features portion. That starts in the beginning. And append it; to that list of
values of feature. Then you return everything. And you need to take the
column names; from your data frame which was TT. And then you take the new
variable you're storing everything as. From this function. Which would be DF to
Nested JSON. You're calling the DF update, and the column names. And then you
get an error.
From something, which would be? Let's see what's wrong here. Oh salary, is it like this yes?
Okay and then we see what that looks like.
Perfect okay, so our first two entries will we take our candidate and then we
have our first name, last name, and all of our information. Cool, now we can do
the same thing with new entries of non nested. Okay, this is the non nested
entries portion. Because, if you notice here :we didn't create a new a new hire.
Like you did right here. And everything is the same
you just didn't add that. And so you can see what that looks like here. So now if
you pay attention. It just gets added into here. You have your salary, your
healthcare, and the date they are hired. Versus what it looks like here with the
nested. Which will call your your hiring date, and then a list of values for those.
Okay ,so that's the that's the difference for that. Now what I would like to say
finally, is your consideration for timing your operations. You should take this
into consideration: for example the with open is faster; than using just a regular
panda dot read. Also, you should notice here, there's quite a bit of a difference
point zero one one seconds versus zero zero five seconds. There's quite a bit
of the speed-up. If you're using large data you need to start looking into
these considerations. Because, it affects the timing considerably. As always
thank you for watching this video. Please, subscribe and turn on that notification
bell. Throw me a like if you thought this was useful content. If not through a
comment and tell me why and give me some feedback. So I can make this channel
better for everyone. Have a good ones see you in the next video. Bye