Splunk has 100s of apps that extends its core capabilities.
The Machine Learning Toolkit App delivers new ML specific search commands, visualizations,
and other tools for performing machine learning to your Splunk data.
In addition to all this, we have ML Assistants that guide you through the process of building
custom models and operationalizing them on the Splunk Platform.
In this video, we’ll explore the “Predict Categorical Fields” assistant, which helps
to predict the class or category of an event using other fields in the event.
This task is commonly referred to as classification.
There are countless uses for classification, from predicting customer churn to the success
or failure of any event.
Note that most of the input and output panels in this assistant have tooltips, giving us
more information about the particular panel.
Hover over a panel title or field label to reveal the tooltip.
For this demo, we will use the Security domain’s firewall traffic dataset that is packaged
with the app, where we will try to predict the presence of malware.
Using the search field, we first load the dataset to get the ‘Raw Data Preview’
panel below, which shows the fields in the dataset.
Our next step is to create model for which we need to select the Algorithm to be used
for classification.
Splunk’s ML App includes several classification algorithms, each of which are better suited
for different kinds of data.
You should experiment with the options to see which one works best for your data.
For this example, we will use Logistic Regression.
Next, we will create model for which we will select ‘used_by_malware’ as Field to Predict
and then for ‘Fields to use for predicting’ we will use bytes_received, bytes_sent, dest_port,
has_known_vulnerability, packets_received, packets_sent and src_port.
Once the “Field to predict” and “Fields to use for predicting” are selected, we
split the data into training and testing set.
We split our data into two sets because we want to test the trained model on a set of
events the model hasn’t already seen.
Let’s pick a 70:30 split to withhold 30% of the data for testing.
We also have an option to fit intercept in the models equation or remove it.
To remove intercept, we have to be sure that when all the predictors are having value zero,
then to be predicted value should also be zero.
For example, in case of mobile logs when the battery percent and no. of apps open is zero
we can say that screen brightness would also be zero.
Therefore, intercept fit can be unselected.
Note: Do not unselect the fit intercept option when you are not sure.
Next, we have the option to save this model as per a naming convention that we can find
on the “Load Existing Settings” tab; we’ll revisit this tab later in this video.
Once we click the Fit Model command, the classification model is created.
Results of applying the model to the test data are loaded in ‘Prediction Results’
panel next to the actual values.
We also get a Confusion Matrix, which is commonly used to validate a binary classifier.
In this matrix we have four blocks:
True Negative is the no. of events Predicted No and also has actual value No.
False Positive is the no. of events Predicted Yes but has actual value No.
False Negative is the no. of events Predicted No but has actual value Yes.
True Positive is the no. of events Predicted Yes and also has actual value Yes.
Confusing isn’t it?
Remember this, True Positive and True Negative are the correct predictions and False Positive
and False Negative are the incorrect predictions.
A better model has small percent in False Positive and Negative.
Sometimes our priority is to get either a lower false positive rate or a lower false
negative rate.
For example, in our case failure to detect a malware can cost a company a lot of money
compared to the loss incurred due to unnecessarily investigating a secured network that was predicted
to have malware.
So false negatives, which are the no. of predictions of malware usage we missed, should be minimized.
This assistant computes some statistics based on the confusion matrix where we can check
veracity for classification based on false positives, false negatives or both.
If we want fewer false positives, then we should concentrate on increasing the precision,
but if false negatives are more important to us, then Recall should be maximized.
Once we get acceptable statistics results, classified malware results can be shared with
the Network Security Administrator for further investigation into the network to search for
malware.
On the “Load Existing Settings” tab we can compare models trained with different
parameters using statistics such as Precision, Accuracy, Recall, etc. and reload the one
that worked best.
Once we are happy with the validation statistics, we can set the training set to 100% and deploy
our model.
We can also apply our configuration to new data.
These assistants not only walk you through the steps required to train, validate, and
deploy a machine learning model, but all of these tools can be reused and provide a variety
of options for operationalizing our model.
You can view the SPL commands used at every step, schedule the configured setup of the
model, create alerts based on the output generated or use one of the validation panels on your
own dashboard.
To learn more about the Machine Learning Toolkit, including the other Assistants, browse through
our ML videos.