Welcome everybody. This is Mr. Fugu and you're watching Mr. Fugu Data Science.
Today, we will be working with nested data using Pi mongo (pymongo). We will make some
use basic queries. show how to do aggregation. and MapReduce. The code will be on my
Github with a link in the description below.
Make sure to subscribe and turn on that notification bell. Like this video; if you
don't like it. Please, leave a comment and let me know why: before you throw a
thumbs down. So we need to import PI Mongo as pym, import pandas, optionally
you can import pretty print. and we will need to import from bison (bson)
code import code. Now we'll set up our connection: where we're calling it client
word equals py m dot Mongo client. and then we're calling our localhost here. My
database is called Berkeley. The
collection has recruiter clients. I am storing that as candidate user info. All
right now, if you would like to see where I got this data for our database. Check
two videos of mine: one, nested JSON example. Where I created the data set. The
other video: where I used it in MongoDB for my little short series that will be
in the second video of the series and I will put a link to those videos as well
at the top of the screen. Now, what we have here is basically: if I was a
recruiter looking for candidates and I have each candidate where they
have a first name, last name, a list of their skill set, the state they live in,
the specialty, the level of experience and are they willing to
relocate. Okay, and then I say I want to find anybody
with the first name of Brenda, sure why not.
So we iterate through our database and find this. So we say candidate user info.
Which is storing my database dot collection. and I'm
saying since, this is nested I have to do the dot method. So candidate dot
first name. And always remember outer dot inner, outer dot inner. okay, so we find
that there's two people with the name of Brenda. All right wonderful ,not very
different from what we would use in native Mongo shell.
Now let's find only people with a skill set of spark but, that's ignore printing
out the object ID. Basically, think of it as your primary key for your
document. Okay so this is only people with the skill set of spark. If you
look here. So we can run some basic queries. That's fine and dandy. Now let's
see if we could use some operators. So we iterate through this and we're looking
for anybody who lives in California or Florida. So let's just pretend that my
boss tells me we have two jobs coming up; and we need people who live in
California or Florida. And let's see if they'd be interested in this job. So if
you notice here: we have two put a string around our operator. Whereas,
if we were using Mongo, native Mongo. We wouldn't to have to do this. Okay, so if you
notice here, this is what the we'll put in the terminal, and we don't need it
here. Otherwise, everything else is the same, okay.
So we iterate through this, and I say well what does this look like? And here's
what it looks like this, with the pretty print. Oh no, this is not. if
I wanted to access these candidates. I have to iterate through this now. and go
inside to plug everything off, if I'm going to either extract this data or
place it into a data frame for instance. We end up doing this query storing
this, as a list. Which I said store California, Florida. Iterate through that.
and then I call a new list that I need to append: all of my candidates. I created
another list that's only my IDs. and then I create my data frame. and after I
create that data frame; I basically, add a new column called
IDs. Where I take the stored list of only my IDs, and it print that out. Here's what
that looks like. so we have our object IDs, and all of our
other information. If you want and you can re sort this. Where this
column was first, if you wanted to. and I showed how to do that in another example.
I think it was in the first Mongo video. But, don't quote me on that.
Now, this that we want to do an aggregation now.
So we set it up very similar to native Mongo in the shell. But we're
still calling our aliased DB dot collection name. Which is
candidate user info. and then we call the aggregate. But, what you have to pay
attention is; each operator needs to be enclosed by a string. So I'm doing a
project, and I say I want to project. I'll the zoom in a little bit more. So I want
to project all of the candidates by first name, that I want to use.
I want to use all of the candidate last names. Now I'm creating something new or
I'm storing what I want to do. I'm calling it number of skills, and I need
to enclose that in a string, as well as this. Because, it's an operator and I'm
calling a condition. And this if needs to be stored in a string, as well as then,
and size because that is a operator as well. So what's going on here? I'm calling
a project: where I'm taking the first name and the last name of what I want. I'm
storing something new, called number of skills. Where, I'm creating a
condition with an if, else basically. Where I say: if we have an array which is our
candidate skills. Then, let's get the size of that list for our skills. If that's
not there; then we're going to just put an n/a. And
that's the Skip any ID tags because, we don't need that. So now we iterate
through that, and what we get is basically; each candidate by first and
last name. Where we have account of all of this skills that each person has. Okay
so some people have a lot of skills some people don't have a lot of skills but
that's okay. Because in life, some people have a lot of skills. Some people don't.
Some people are really specialized in one thing, you know.
Now lets try something really simple. where we do a match; and we look for
anybody who has a skill or excuse me a specialty of machine learning. Since,
that's a really hot topic nowadays right. and the only thing you're doing
different here is you're enclosing this with a string. Where we're saying let's
use the aggregate, and let's take a match. Similar to if we're doing a regular
expression match, and find anybody with the specialty of machine learning. and
let's just print them off. So we got somebody with the last name of Stuart
who does machine learning, someone with the last name of Lin that does machine learning
and so on, okay cool. So that means that we could do aggregations and queries. But,
now let's run MapReduce. and this is kind of interesting
what's going on with MapReduce all right you need to import this I did it at the
top. But this is really important okay. So you're saying the from by son (bson) code
import code with a capital C okay. What's going on here is you're creating
your map function, you're creating your reduce function, and you're throwing it
into your Map Reduce, right. so what are we doing here? I'm creating a function
where I say emit, this dot candidate state. So the emit is always with your
map function, If you've never used map functions. and this you have to use.
So your basically saying, I'm calling this, is basically
going on. This dot candidate dot state. and I'm counting each state, and I'm
throwing in the reduce function. Where I'm going through the state and the
value, and I'm starting with a count of zero and I'm iterating through the
length of my basically the length of all of the states. and as long as there's
some you know as long as there's something there, I could iterate through
it. and I and then I increment my count and then I return it. sorry
maybe I should zoom in just a little bit to help this a little bit. and then
what's going on here is you still use the JavaScript variable notation here.
But, you have to enclose all of your functions with strings. so basically what
you need to remember you use the code you put the parenthesis. You put the
quotations, and then whatever your function is on the inside. When you
actually, call this and you say I'm going to call my Map Reduce. You still use your
collection alias. But, here instead of Map Reduce being map reduce one word and
the R is capitalized. Here it's not. It's map underscore reduce. and then you call
those in. And what's going on here is I'm storing a file; where it's called skills
by state. Okay, if you were in the Mongo shell it will look a little bit
different. okay let me show you that so in the Mongo shell: what would happen is
you...
where's it at? Is that it? Wait that's Aggregate. Oh it's a different example.
I'd have to open it up. so lets open that up right here, okay. So what you're doing
is here right here you're calling out. So you're saying this output is data into a
file. Instead of doing it in line and it prints everything and on the screen. and
then when you're actually calling it. If you were in that on shell you would do
it this way, where you're calling that file. But, it's a little bit different in
pi mongo (pymongo). So here I can do it two ways. I can say I'm calling the function
that I I just created here with a MapReduce, and I do a dot find and
iterate through it, and then append it. and this is what it looks like. Where I
have each statem with a number of clients for that state. For my recruiter database.
and did this example to say what if I'm looking for bumping up the number of
clients that I have in states that are really poor. Like. low numbers. Okay, so you
have to do a sorting here. So said all right, let's do a value dot count. In this
sort and find out which states we need to get more people recruited basically.
In this example what I did is just like the MongoDB version. I did DB dot skills
by state. and that's what I called it here. and I did dot find, and I sorted it.
and I iterated through it, and it pulled up the same data. Whereas, here I'm only
calling it directly. So I'm basically saying DB collection name,
map reduce, and calling it and running it directly. Instead of this file that I
created. and this file is shown here if you go into Mongo and you look. We can
see right here. Let's see if I can make this bigger. Right there. and
that's where it is. So that's the file that's created. And it's actually on in
your collections and that's how you're able to call it with the DB find or DB
dot skills by state. Okay, so that's some examples that you can use with PI Mongo.
I would like to say thank you for watching this video. and I hope you like
it. if you do not like it. Please, leave me a comment before you do a thumbs down.
and as always if you subscribe turn on that notification bill I'll see you in
the next video thanks for watching. Bye