Welcome everybody. This is Mr. Fugu and you're watching Mr. Fugu Data Science.
Today, we will be creating a nested dictionary using JSON formatting, for a
fake data set. As always the code will be linked in the description below. The
objective today, will be to create a nested JSON object, understand the pitfalls when
you're using data frames to create these, evaluate data frames that are dumped as
CSV files in either JSON or dictionary formatting, then we will create a
function to properly store our data. and then describe the format and ability to
send it to Mongo if needed. That's optional so we're going to use faker,
pandas, random, request, beautiful soup, regular expressions. Okay this data set
is basically, assumed that you were in the position of being a recruiter and
you were looking at people by first, and last name, who had particular skills, they
had a state that they lived in, a specialty, and their willingness to
relocate or not. So here's our skills list that I created. We have this
function where we're creating random skills; where we're generating a seed
giving us data consistency. We're going to have 500 people. I put this here
because we're generating random integers. Where you're going to have at least one
all the way up to the length of the list. I did this because I wanted to have
a person one has at least one skill all the way up to all of the skills in that list.
and then we append that into our list and that looks like this. So you
say the first person and list has six skills, next has six skills, next one has
one skill, etc. Then you iterate through that list and
then you have to use the extend so it's not nested. and you say okay lets
randomly sample all of the skills, and lets enter the number of skills that
each person is going to have. So then you create that list, and you know return it.
And here it is you have six, you have six, one, and then eight. So that works out.
Perfect next, thing we want to do is create another random seed. So we create
our names,; so we're having 500 names. We're saying okay, if we're trying to
create non duplicate names. But, you can have real duplicate names in
the real world. But, in this case I didn't. So we're going to split those on a space.
Because, in the original string here is creating a string with the first, and
last name split on space. But, I wanted to split in on a space. So then we have
each one of them separated. I got a list of the US states abbreviation CO, CA, FL you
know etc. So I went on this web page use some web scripting to practice my skills
so I don't forget. Found that it was in this particular div column. Everything
was stored in a table formatting. This is the attribute or the class color BG
Green stored all of our information. Then we went into the tr tags make sure to
use UTF formatting, split everything on a new line. Okay cool.
Ten we take our tags and say okay let's only look for the tags with the strings
having a capital A through Z and only taking if they have two back-to-back
letters. So then that delineate your you know your state abbreviations. Then I
just parsed through and replaced anything I had a td tag and just replace
those with with basically empty space. and then here we go, here's our random
list of states for our 500 people. Then I did a random dot choose. so like N choose
K situation. and you do 500 of these that you want with our state abbreviations. so
oops okay so let's find out. Oh that's way
okay here we go here's our people, and their random states. Now are these people
willing to relocate: yes, no, or maybe. Then we created 500 of those again with the
random map choices. Now we take our specialty here and our experience level
we do the same thing and I said okay let's try to create a list of tuples
what we're saying alright each tuple: I'll show you real quick has our
specialty by name, and then what their specialty is. So here it is specialty
machine learning, specialty data visualization, are cool. Then we're
creating a dictionary formatting here for each of those. and it looks like this.
Specialty data bases same thing you'll have for the experience levels. Then you
could say alright this look at our just look at them together. If you zip those
together. and here's what it looks like so tuple of dictionaries, okay.
Now I'm taking the first name in the last name zipping those together with
all of our data that we generated. and this is what the first two entries look
like: first name, last name, skill set, where they're from, and then experience
with their specialty and are they willing to relocate. Then create a data
frame with all of the column names and dump this is as a file.
and show you the head of this. Here is our tuple dictionaries here. and then
here's the rest of our data and it looks pretty good right?
But, then this glowing this look at what this file looks like. And here's where we
start encountering some of our issues okay so we dang it okay this just dump
this file. So maybe I store it somewhere else
Oh
job seekers it's not txt its CSV oh
because I got this in a different okay there it is all right here's our header
right here and since we have a nested dictionary as storing these is entry
zero. and then it has all of your information with it separated by your
new line. Here's the next one separated by a new line, fine. But look at that so
now you have your list getting stored as a string well normally this is an issue
you could store this just like this and then import it back into a data frame.
And that's just how the formatting natively works. But, if you would like to
preserve your data and not have this happen. and not have this happen again
here, for the other one. For this one as well. You have to either store it as a
pickle. But, then I'm not sure how you're going to import it into other things.
Here, I was using this data set for a importing and exporting through MongoDB. and
this formatting doesn't work for me. when I'm calling nested methods. So how do I
rectify this right? I create a function that actually does this for me. Okay. So
I'm calling this. We're gonna start with candidates we'er taking the list with
everything that we have, all of our columns ,and we are creating an
un-nested junk. So here we go so we create that and now to let us start with what
we need to do. So nested JSON function where we're taking
a data frame we're taking our candidates and then I just I just abbreviated this
chopped it up but we're throwing in states right. So you take your you take
your JSON variable where we're taking a dictionary where we have features and an
empty list where we're storing everything. and we're iterating through
our dictionary. And for a variable feature; we say inside of features we
have candidates, which are dictionaries. We iterate through each of our
candidates and we store our key value pairs and then we append our dictionary
features with a list of features that we just created. Ok so after you do that you
need to take your column names from your data frame you need to put
that here. and then what you're calling in. Which is
my un nested candidates data frame. and then now it prints out like this which is the formatting I
was looking for. Where I have candidates where this is the first: just do this so
you can see what it looks like from the beginning. So you've got to start with
features. That's what I'm calling here and then everything else is nested list.
So then we take our list of nested dictionaries. where we have candidates and
inside of candidates: you have first name, last name,
specialty, experience, relocation. You don't have that string string junk
in front of this, in front of this anymore. So you're good to go
right? If you wanted to take this then you can send it into MongoDB. You can
send it into other databases ,you could manipulate, or play with it, do whatever
you want. But, today you learned how to create a
function that creates nested dictionaries. You can also take this out
and, oh I change this up I'd have to add it in here but then you can use
something like this and then mess it even further. But, nevertheless I was
going to show that. But, let's the end of this video as always please like and
subscribe. and turn on that notification bell. So
you can see the next videos coming out. Thank you, for watching have a good one
Bye