Hello.
Welcome to this RapidMiner Radoop Tutorial.
In this tutorial we will briefly describe how to connect to Hadoop using the Radoop
extension and then we will show you how to create a prediction using the Radoop operators.
First thing to do before you get started with Hadoop is to install the 'Radoop' extension
from the 'RapidMiner Marketplace'.
Please check out the video on extensions for detailed instructions.
After the installation you will see an additional folder called 'Radoop' in your 'Extensions'
folder of the 'operators' panel and also the additional 'Hadoop Data'-view.
For this demonstration we are using the Hortonworks sandbox.
The 72 operators coming from the Radoop extension allow you in essence to seamlessly use a variety
of functions for a number of technologies such as Hive, Apache Spark and Mahout to prepare
and to model your data.
In order to access the data we of course need to establish a connection to our Hadoop cluster
or in our case to our small sandbox.
The details on connecting to a cluster are beyond the scope of this tutorial but we still
want to briefly touch on the different elements of the setup dialog.
To see the details of our existing connection click on the 'Connections'-menu and select
'Manage Radoop Connections'.
You can see that we have a connection setup here called 'Hortonworks Sandbox'.
I'll select 'configure' to give you a quick walk-through.
You can see below the name of the connection and the option to fill in information about
your specific installation.
If you expend the 'Hadoop version' drop-down you can see the large number of platforms
which we are supporting.
In the next block you have the 'Master settings' where you can also add Multiple Masters for
larger installations.
I have filled in the local IP-address as you can see and left everything else at the default
to match the sandbox settings.
In the SQL-settings I have adjusted the default to be set to the latest Hive-Server version.
Of course you can also add your security settings as for example your Kerberos setup or others.
For this demo we have further adjusted the Spark version and the location of the Spark
library and set one additional default Spark parameter.
After you have adjusted your settings we recommend you execute the 'Quick test' and the 'full
test' to ensure everything is working properly.
Before we start creating a process let's check out quickly the 'Hadoop Data'- view.
This view helps us to assess the data which is available for us in our cluster.
We can retrieve the metadata by double-clicking on our connection.
Once loaded we have a nice overview of our tables.
If we click for example on our 'titanic_training' table we can see what attributes it contains.
You could even retrieve a preview here if you need to take a quick look at a sample
of your data.
Now let's go back to our design View and start building a process with Radoop.
We will drag and drop in the most important operator which is called 'Radoop Nest'.
You will find it all by itself on the top level of the ‘Radoop extension’ folder,
emphasizing its importance.
The nest operator causes everything to happen on the cluster, instead of on your local machine,
which is what you want in order to use the power of your cluster.
Let's connect the 'Nest'-operator to the results port.
By doing so, we are actually allowing information to be brought back to our local machine for
display in the 'Results' view.
However, it will always only be a sample to not clog your system.
You can tick the "change sample size" check-box in the 'parameters' panel in order to see
the sample size and adjust it to your needs.
This is, by the way, one of the options to pull data from your cluster into your local
environment and work with it there, using the standard RapidMiner operators or even
in case you need to do a data-dump.
Keep in mind that the sample size will always work like a hard-coded filter.
Now let's double-click the 'Nest' operator to enter the sub-process level, where we will
configure a simple prediction on the titanic incident.
For that, we are going to use the same data which is available in your RapidMiner Studio
'samples' folder - just in case you want to follow along in a non-Radoop fashion.
We will start by using the 'Retrieve' operator from our Radoop extension.
The name is in this case a bit misleading as we are not really retrieving the data but
it remains in Hadoop, as mentioned before.
When we click on the table drop-down in the parameters Panel, we can see that we are now
connecting our sandbox with the available tables so the meta-data has been gathered
here for us here as well.
The table we want to use now is the "Titanic Training" data.
To show you what is in this data we can simply execute the process.
You are seeing an indication that it is running and finally we get the data displayed because
we had previously connected the 'Nest' operator to our results port of RapidMiner Studio.
In our case it is really all the data as our table is with just a bit more than 900 rows
- or examples as we call them in RapidMiner – a very small dataset.
Looking at the data you can see some of the known information about the passengers of
that famous cruise.
We do have most importantly a label - indicated here in green - which tells us who survived
and who didn't.
Also we have some further information about the passengers like: Age, Passenger class,
gender, some information about family members on board and the price they paid for the voyage
which we want to use to build a predictive model from.
Now let's go back to our 'design' view and into our 'nest' operator.
The second thing we are going to show you is a simple validation.
We will use a 'split validation' operator here - again make sure to use the one from
the Radoop extension folder.
We will do a 70 to 30 split on our data.
Now within the 'split' operator we have the two sides, one for 'training' and one for
'testing' the model.
We will populate the two with a 'decision tree' model, an 'apply model' and a 'performance'
operator.
Which we can drag and drop conveniently from our 'wisdom of the crowds' section or from
our ‘Operators’ panel.
For the decision tree you can see the 'Spark' icon and for the other two you see the 'Hive'
icon displayed on the operator.
So for the Nesting we use standard Hadoop technology and for the other features we are
using as mentioned other technologies such as Spark and Hive - all that just by pulling
in the RapidMiner Radoop operators.
We will leave all settings as default, except for the decision tree where we will select
'entropy' as criterion as that is better suited for attributes that occur in classes.
If you would like to learn some more on, models, validation and performance interpretation,
please refer to the respective video in our 'Getting Stared' series and also to the '5-minutes
with Ingo' videos, which you can find on our YouTube channel.
In this demo we will just continue to connect all the relevant ports in the 'training' and
in the 'testing' section, so we can create and assess the quality of our model.
In the last step we will apply the model to some unlabeled data which will conclude our
prediction process.
So let's move up to the relevant level to retrieve the unlabeled data.
The unlabeled data has no information about the survival of the passengers which it contains,
so this is what we want to model.
To retrieve it, again we need to use the 'Retrieve from Hive' operator and this time we will
have it pull up the table called 'titanic unlabeled'.
We are going to apply our model to the new data.
For that we have to use again our 'apply model' operator.
Remember to use the one with the Hive-icon!
So now we can connect the 'out' port from our second retrieve operator to our 'apply
model' operator.
We retrieve the model from the 'mod' port of our validation operator and we send the
information to our results.
We will output the, now 'labeled' data, the model and the performance from our validation.
This means we have three outputs which we need to account for also in our 'Nest' operator
if we want to see everything displayed in the 'results' view of RapidMiner Studio.
So let's go one level up and provide three output connections.
And as we execute this by hitting the 'Run process' button, you can see the progress
on the execution indicated here by the icon.
In the bottom we get an information displayed what is currently executed.
We are now in the validation.
If we access the Hadoop cluster via our web-browser we can see that a job has been submitted and
is status undefined.
If we give it a bit of time it will finish, so let's go back to RapidMiner Studio in the
meantime and have a look at the results.
The model we generated is a decision tree which is showing us that gender was the most
important attribute to determine survival.
So the "women and children first" etiquette was actually followed.
The next important attributes were - not surprisingly - of a less noble background.
It was 'passenger class' and 'price of the ticket' which determined the faith of the
passengers.
If we look at the output of our performance we can see the overall accuracy and finally
we can then see the results of our prediction on the unlabeled data.
It has been amended with a predicted label together with the confidence values.
So this concludes our introductory tutorial on RapidMiner Radoop.
The most important items to keep in mind is that, everything needs to be nested in order
to have it executed on the Hadoop cluster instead of on your local machine or RapidMiner
Server, and the Radoop operators need to be used to execute the relevant steps.
Of course there are options to use the RapidMiner Radoop operators in conjunction with the usual
RapidMiner operators but this is beyond the scope of this brief video.
Thank you very much for watching!