Welcome everybody. This is Mr. Fugu and you were watching Mr. Fugu Data Science.
Today, we will be doing a small tutorial for MongoDB with MapReduce. As always the
code will be linked in the description below. The objective will be to give a
basic idea of what MapReduce is, some of its use cases, create a few examples with
explanations, and show for instance when aggregation might be used in place of
MapReduce. In general MapReduce is used for a
parallel computing for large data sets. Where you're processing the data either
in a file system or a database. The file system is taking unstructured data.
The general idea of map takes in your local data and creates a temp file and
makes a single copy of the data to store. You have a shuffle step where you're
redistributing the data, and then you're reducing by processing the data by a
keys in parallel because, you're creating key, value pairs. Now you are writing
these functions in JavaScript, so just take that into consideration. Okay, so the
map; just think of that as mapping multiple values to one key all right.
Then we will be using what's called an emit function with our key, value pairs.
and then you have what'll be a: this dot and that is to work with whatever
document you want a process for Map Reduce. And then just think of the
reduce: where you're taking a single key and mapping it to a list of values. Okay,
so some use cases in the real world: you can use this for graph theory,
classification, inverted indexes, machine learning, document clustering are some of
the examples. Okay, here's something that is
kind of overlooked. But, if you're doing a reduce function and you're calling a
more than once. Then you need to take into consideration that it needs to be:
associative idempotent and commutative. Okay, so from the data last
time we have this. Where I created a database: where I'm saying I'm a
recruiter and I have candidates and these candidates have this information.
Where it's a first name, last name, what their skills are, what state they live in,
their specialty, their experience level, are they willing to relocate. Okay the
first example today we will do is: assume that your boss tells you hey, let's find
all of the candidates based on it on each state and let's count them. All
right, and your boss wants you to get more candidates for that for your
database. You know they want you to generate more revenue and find out what
states are basically lacking. So we need to create a variable and here I'm
calling a map funk. You could call it whatever you want to call it all right. So
we're going to take in our map funk, and saying alright function open parentheses
because we're not calling anything in there. And then we're using our emit
that I mentioned. and then this dot candidate dot state. so I'm saying this
is what I'm calling. I am calling the state, and I'm counting that. So we plug
that in for map function. And then reduce we're taking our state's and our value
that we're counting. and we're iterating through that through the length here of
it. Where we increment our count and return the values. Okay so we can plug
that in here. So let's do this so we have basically a clean slate to look.
and then I can see what's going on here. Oh this is from something else I was
playing with. Okay so let's take this make sure it's copied pasted in okay so
there now we have our MapReduce function. okay. Now, we have to take our reduce
function. That I just explained. We plug that in if I could get this stupid
thing to copy. Perfect then we have to call Map Reduce. So what happens is you
take your DB dot recruiter clients, which is our collection. and you say Map Reduce
and inside Map Reduce you have to tell it what you are mapping. So you do your
map function, your reduce function, and then out it's a file that I'm creating.
Which are the skills by state. You could also have a different alternative here
which is in line equals true or one. Which would mean you're printing all
your output directly but if you have a large data set you don't want to do that.
Okay, so do that and we wait ,perfect. So we took in 500 of our clients, we emit
a 500, and we reduce it to 50. Which is their number of states, and our output is
our states and okay, true. Now we're gonna say that's okay, I could do this.
We're going to now call, what I just named this here. so we do that. and
then we have our count. so we say DB dot and then the name of the function of the
file that I just created. Then I just wanted a prettier format.
so then we say okay Alaska has ten people in the in the database.
California has eleven, Florida has eleven, etc, fine.
Well let's sort these values. And find out what we need to adjust our database. so
basically I said sort equals one. Where it's ascending, where as if it was
negative one, its descending order. So it looks like Vermont, South Dakota, Virginia,
New Hampshire, and Indiana and a few others less than ten. So we'll just say these
four right here are basically where we need to bump our
values up, and get some more clients into our database.
Okay then for the example two: assume well your boss is interested in what
frequency of all skills. Okay so this let's see what that really looks like. So
I'm taking the skills. And the skills are a list of values. All right so I do this
dot candidate dot skills. Because, it's nested and then I'm iterating through
that, and I'm emitting this skill. and I'm going to count those. okay so we do that.
Then we have a reduce function; we're taking our keys and our values and we're
counting. we're counting our values and we're incrementing and returning the
count. That's what's going on here. Then we need to do our DB collection
dot Map Reduce. and then we need to call our functions in that we're looking for
again. Because, we want in the Map Reduce to say all right this is the map
function you're looking for this is the reduction function you're
looking for, and then output will be stored in skills underscore stuff. So we
take that and wait for to calculate, cool. so can 500 entries and then this looks kind
of strange: but, remember it was a list of values in each list had different values
from 1 to 9. So that's why if Tom 2469 then we reduced it to 45, and then the
output will be 9. Because there were up to 9 values. so now we call the output
file that we just stored. so DB dot and then the files name, and then call that.
and then perfect so that there were 9 skills and then we counted the number of
times that the skills happened. okay so assuming that like your boss a hmm
what would people have what skills? okay or how often do particular skills come
up? So then that's what's going on here, and it looks like it's
distributed you know fairly evenly. you know which is interesting. Then I said
okay, what if we wanted to do something else? So if I was looking for the first
and last name of somebody and trying to find out are they willing to relocate?
Well that's a little bit difficult to do a Map Reduce. Because, now how am I going
to set this up. Because, now I have basically two keys and then my value is
is my relocation. Well now produced that might be a little challenging you could
probably do like a string concatenation or something like that to combine the
first and last name probably, and then run this. But, here I found it's just
better that I do an aggregation. Where I do a projection: and take the first and
last name, then create like a temp variable basically. Saying relocate yes,
and then ignore the ID's. Let's see what this looks like. So then here we're
saying okay first name, last name, and then are they willing to
relocate: false, true? Okay so this person is willing to relocate, this person
willing to relocate. okay that's a little bit along the line of what I was looking for. But lets
see what else we could do. so what about if they want to relocate
yes or no? Well then we need to use that or operator so we have kind of similar
to what we're doing here but we're expanding onto it with the or operator
for yes and maybes. So then we say all right let's do an aggregation where
we're projecting our data; for the candidates first, and last name. We
basically have this new variable name let's taking in to consideration if
someone is willing to relocate or not and printing that out is true. Okay, so
we're gonna kind of closer. What else do we have? Well we have this one
that I mentioned here: it's kind of a weird format, you're gonna see why.
So you're doing the same kind of thing and expanding on the last one :where
you're saying let's aggregate this, project let's take our first and our last
name. Let's look for the relocation but now we're doing a conditional statement.
Where 8we're saying, if we have an a relocation equal to no. Then this remove that. But,
what's going on here? Otherwise, just depend all the other information. Well
this seems like it's gonna be kind of funny.
What is it removing? Well as removing the whole relocation part. So basically for
your eyes it's pleasing: and you say okay yes, maybe, maybe, maybe, yes, yes, maybe. Okay
cool and then this is just nothing there okay well that might be something you
want to look at. and then what happens if I just just something really simple and
do a match. Okay well this do a match. Oh
yeah, that matched a bunch of things. It matched everything I had yes. So you
matching the whole entire document, as long as it had a relocation of yes. So
what's the deal here? Basically, if you want to use
map reduce. Consider using it in cases where you have a lot of data that can be
spread across many machines for parallel computing. That's where it really strives.
If you have little data, I would suggest doing other methods. Also, if I'm running
something where I'm trying to count count values, doing average, things like
this. and I could do it with the key value pair then the MapReduce would
probably be a good idea. If I have multiple keys and values I'd have to
really think about that and it would probably just run an aggregate function
like I did with these. But, as always like subscribe and turn on that notification
bell. and I'll see you in the next video Bye