Welcome everybody, you're watching Mr. Fugu Data Science.
Today, we'll do MapReduce once again with pymongo. As always the code will be in
the link in description below. The outcome and purpose; create a connection from
Python MongoDB, generate a MapReduce example with and without a query, export
CSV or JSON from MongoDB. If you're unfamiliar with what MapReduce feel
free to pause the video and read this to get a general idea. Now install PI Mongo
(pymongo) if you don't have it. Here's three options you can use right here. Depending
on if you're using a anaconda or pip. Now import PI Mongo, pandas, CSV, JSON,
as well as from bee son dot code (bson.code) import code with a capital C. Now to establish a
connection; you call PI Mongo dot Mongo client. And then whatever you're
connecting with. I'm using local host because I don't have a dedicated server
website. The database I'm going to use is Berkeley and the collection; think
of table is recruiter clients. These data look like this. It's nested where
everything revolves around a candidate. Each candidate has a first, last name
list of skill set, state they live in, specialty, experience level, are they
willing to relocate. I will generate two ways of outputting your data. One will be
iterating through the variable name, two calling the output file you generate
which would be people underscore skill count. Now, let's create the map function/ I'll call it map func as the variable name and then you have
to use code quotation and then we're creating a function without any
arguments. Where I'm looking to create a variable named skill; which will be my
skill set. So I'm taking anything that you would like to call directly will
use: this dot. So this dot candidate dot skills, okay. Iterate through these skills.
Then I would like to use what's specific to the map
function. Which would be called emit. And then the emit I'm generating, a key
value pair for the name. We're wanting to use this dot
first name. And then I want to basically just think of concatenating strings. I
want to combine the first and the last name columns together. To be one, okay.
Then we need to call the skills that we just created above; as well as I want to
take the new key value pair of skills. Where I'm calling this dot candidate dot
skills again. And then i want to do a count of those skills. So what's going on
as the output the end? I will have the first and last name together, a list of
their skills, and a count of how many skills they have. Now let's creates the
reduced function. and use the code once again. And this will take arguments for
our function. We're going to use the key as the name and then the value is the
skill count. Then, what you need to do is return a value which
will be our array, dot sum of the value for our skill count. So what's going on
here? Since, we have an array were using we need to use array dot sum. So
it could actually take account of what's going on. And to bring it all together we
will call our recruiter clients collection; and then use
map on the score reduce. Which is a little distinct from native shell Mongo.
Then we'll call our map func, we will call a reduce function, and an
output file I'm creating called people underscore skill count. Then
to access all of this, I am going to print it out. But, in order to do it for
the first example I will iterate through this. So I'm iterating through all of the
skills for Map Reduce. And we're going to do a find, like you're doing a query.
We want to append our skills and the people's names and then I'll print off a
few examples, so we could see that this works. We have the first and last
name which are combined now. This person's skills and counting the number
of skills. An alternate way of doing this without iterating through; is directly
calling the output file itself. The output file is actually stored as a
collection, okay. So if you look right here, this is the collection is stored in.
We can call it directly. So that's what we're going to do now. Then what
you do, is say
I'll just call it my doc equals DB dot and then people skill count dot find. And
let's find everything. Then let's create something called people as
a empty list, and then I'll just iterate through my document of everything. And
then we need to append this, and you see what this looks like. And it should be
the same. There we are. So you can call it as a collection and
call the stored output file that you just created; or you can iterate directly
through what we just created and not call this just appending. But, it's useful
because you could run queries just the same. So then the store this as a data
frame now. I'll just make up some list names and then go through the data for
our people skills.
Yeah okay let's go through our people
skills. And then we'll say the name of the skills
okay so we'll look through the name of the
skills for our data. And we're looking at the ID, and then we're looking at the
skill count of our data. And we're taking the value, okay. And then if you're paying
attention here, you look because it's basically like nested once again. That's
why I'm calling these like this. So we could actually take the information that
we need. Then we need to start appending our skill count, and we need to
take the the name skills. and then we need to let's do a data frame creation
of the people's skills, so here's their skills. But, we need to store this so we
could add the other column that has the count. So Then let's call this people
skills count for our data frame.
Then add a new column for our count; which, would be skill count. And then
that'll just be this. Let's print it off. There we go. So we have our new column
that we just created. Now, we have first-name, last name, all those skills, and
their count as a data frame. Okay, cool. So let's do the same thing with the query.
You could just paste everything what you just used from above and this is the
only bit that we're adding in the end. When you're actually calling the
MapReduce. We put your map function your reduce function the output file name and
then now you can add your query. There's a finalized step that you can also add
but with these data it's not appropriate just because of the nature of it. Okay,
so my query will be kind of interesting and seem funny. But, we're gonna call an
expression where we would like to take anything that's less than or equal to a
skill set with the specific size. Okay, so I'll explain this in a second. So it
makes sense. Let's take the candidate skills; then we want to take two skills.
What you're doing is working inner to outer, it's kind of weird. So doing the
size of the list and you want to take only a size of two. I'm just making up.
What I'm saying I only want people with skill sets of two ;and then I'm basically
saying less than or equal to that value. I could do like three and then saying
less than or equal to, or I could say less than that. Who cares and this print
that off. right Oops
and see where I messed up. So expressions right, this is right. This should be right.
Expression less than
and this happened yesterday. I call this,
size. Oh, right there. There we are. So this is what happens when you're live
coding. Now, what I did is I just printed
off people with I could just say less than. So originally I'm looking at people
with three skills. And then I'm doing all less than. So now it's only printing off
two skills, one skill, one skill, one skill etcetera. It just depends on what
you wanted. But, it's proof of concept for that. Now, let's do a dumping of our CSV
file. Now, I said pay attention here: because, there's an issue. There's
two ways to do this. And you're gonna see what happens.
Let's just do, let me just cut and paste for sake of time. So you can see what's
going on. I'll explain this. Okay, so what I'm doing is, I'm taking some arbitrary
query. And I'm converting it into a list where, I'm taking I'm calling everything.
I do not want the output of the ID but, I want to take the first last name, skills,
and state they live in. We could look in what that is real quick.
Here's what that looks like, okay. Then I am creating a file that has the first
name, last name, their skills, their state. I'm calling it this file name I'm
writing that file. I'm aliasing it as an out file, right here. Where I'm creating a
dictionary with these names and it has a header. And I'm iterating through it or I
could print this. And then that's what it looks like because it's nested still.
Then you have to go inside the nested values here with the second loop.
And you can look at this and then now it's going inside. So V is going inside
of that but, then Y it's just calling the candidates, okay. Then we say if y
equals candidate; which it does. Then write that file of V. And V like I just
showed you was everything that you want, okay.
So then let's look for this file candidate people here and see what it
does. If we import it back in as a data frame it looks alright. But, there's a
problem with this. This file has the header and all the information. But,
if you notice none of this is nested, okay. If you want to retain the nested
original data. You'll just do an import JSON, we'll take the same query
what we did above. And it's really short code. Where you just do a with open
statement. And we'll just call it candidate abbreviation dot JSON and then
we write the file. Use some arbitrary designation and I'll do a JSON dump.
Then we'll call our abbreviated people and then what you're calling it. Then
we'll look at this candidate abbreviated
dump. And if you notice it keeps all of the nested data. So there's two
distinctions of how you want to store your data. Two different examples. As
always I want to say thank you for watching this video, and please subscribe.
Turn on that notification bell. Leave a comment if you thought this was useful.
If not leave a comment and tell me why so I could approve this
channel. Feel free to leave feedback everyone or
just comment and say like the video or say hi. Have a good one I'll see you in
the next video. Bye