Splunk has hundreds of apps that extends its core capabilities.
The Machine Learning Toolkit App delivers new ML specific search commands, visualizations,
and other tools for performing machine learning to your Splunk data.
In addition to all this, we have ML Assistants that guide you through the process of building
custom models and operationalizing them on the Splunk Platform.
In this video, we’ll explore the “Predict Categorical Fields” assistant, which helps
to predict the class or category of an event using other fields in the event.
This task is commonly referred to as classification.
There are countless uses for classification, from predicting customer churn to the success
or failure of any event.
Note that most of the input and output panels in this assistant have tooltips, giving us
more information about the particular panel.
Hover over a panel title or field label to reveal the tooltip.
For this demo, we will use the telecom customer dataset that is packaged with the app, where
we will try to determine which customer are more likely to churn.
This information could help the marketing and sales team to target specific customers.
Using the search bar, we will first load the dataset to get the ‘Raw Data Preview’
panel below, which shows the fields in the dataset.
Our next step is to create model for which we need to select the Algorithm to be used
for classification.
Splunk’s ML App includes several classification algorithms, each of which are better suited
for different kinds of data.
You should experiment with the options to see which one works best for your data.
For this example, we will use Logistic Regression.
Next we will select ‘Churn’ as ‘Field to predict’ and then for ‘Fields to use
for predicting’ we will use CustServ Calls, Day Mins, Eve Mins, Int’l Plan, Intl Calls,
Intl Charge, Intl Mins, Night Charge, Night Mins and Vmail Plan.
Selecting fields, in general, is a hard problem that depends on the use case, characteristics
of the data, and properties of the learning algorithm, itself, that are beyond the scope
of this video.
Once the “Field to predict” and “Fields to use for predicting” are selected, we
split the data into training and testing set.
We split our data into two sets because we want to test the trained model on a set of
events the model hasn’t already seen.
Let’s pick a 70:30 split to withhold 30% of the data for testing.
We also have an option to fit intercept in the models equation or remove it.
Before removing intercept, we have to be sure that when all the predictors are having value
zero, then to be predicted value should also be zero.
For example, in case of mobile logs when the battery percent and number of apps open are
zero we can say that screen brightness would also be zero.
Therefore, intercept fit can be unselected.
Note: Do not unselect the fit intercept option when you are not sure.
Next, we have the option to save this model as per a naming convention that we can find
on the “Load Existing Settings” tab; we’ll revisit this tab later in this video.
Once we click the Fit Model command, the classification model is created.
Results of applying the trained model to the test data are loaded in ‘Prediction Results’
panel next to the actual values.
We also get a Confusion Matrix, which is commonly used to validate a binary classifier.
In this matrix we have four blocks:
True Negative is the number of events Predicted No and also has actual value Number
False Positive is the number of events Predicted Yes but has actual value Number
False Negative is the number of events Predicted No but has actual value Yes.
True Positive is the number of events Predicted Yes and also has actual value Yes.
Confusing isn’t it?
Remember this, True Positive and True Negative are the correct predictions and False Positive
and False Negative are the incorrect prediction.
A better model has small percent in False Positive and False Negative.
Sometimes our priority is to get either a lower false positive rate or a lower false
negative rate.
For example, in our case failure in predicting churning customer may cost a company a lot
more compared to the loss incurred due to unnecessarily strengthening bond with loyal
customers that were predicted to churn.
So false negatives, which are the number of predictions of customer churn we failed, should
be minimized.
This assistant computes some statistics based on the confusion matrix where we can check
veracity for classification based on false positives, false negatives or both.
If we want fewer false positives, then we should concentrate on increasing the precision,
but if false negatives are more important to us, then Recall should be maximized.
Once we implement the model to the whole customer dataset, the classification results could
be shared with the marketing team to implement different marketing strategy for Loyal Customers
and Churning Customers.
On the “Load Existing Settings” tab we can compare models trained with different
parameters using statistics such as Precision, Accuracy, Recall, etc. and reload the one
that worked best.
Once we are happy with the validation statistics, we can set the training set to 100% and deploy
our model.
We can also apply our configuration to new data.
These assistants not only walk you through the steps required to train, validate, and
deploy a machine learning model, but all of these tools can be reused and provide a variety
of options for operationalizing our model.
You can view the SPL commands used at every step, schedule the configured setup of the
model, create alerts based on the output generated or use one of the validation panels on your
own dashboard.
To learn more about the Machine Learning Toolkit, including the other Assistants, browse through
our ML videos.