Splunk has hundreds of apps that extend its core capabilities.
The “Machine Learning Toolkit” app delivers new ML Specific search commands, visualizations,
and other tools for applying machine learning to your Splunk data.
In addition to all this, we have ML Assistants that guide you through the process of building
custom models and operationalizing them on the Splunk Platform.
In this video, we’ll explore the “Predict Numeric Fields” assistant, which helps predict
the value of any numeric field using other fields in the event.
This task is commonly referred to as regression.
There are countless uses for regression, from inferring unknown values to anomaly detection.
We will cover those later in next videos.
Note that most of the input and output panels in this assistant have tooltips, giving us
more information about the particular panel.
Hover over a panel title or field label to reveal the tooltip.
For this example, we want to detect the unusual usage of VPN network using the application
usage dataset.
Unnecessary uses of VPN network bandwidth costs a company a lot of money.
Using the search field, we first load the Application Usage dataset that is packaged
with the app.
After clicking the search button, we get the “Raw Data Preview” panel below, which
shows the fields in the dataset.
Our next step is to create model for which we need to select the Algorithm to be used
for regression.
Splunk’s ML App includes several regression algorithms, each of which are better suited
for different kinds of data.
You should experiment with the options to see which one works best for your data.
For this example, we will use Linear Regression.
In this example, let’s try to predict the ‘Remote Access’ usage based on CRM, Cloud
Drive, HR1 and Webmail fields.
Selecting fields, in general, is a hard problem that depends on the use case, characteristics
of the data, and properties of the learning algorithm, itself, that are beyond the scope
of this video.
Before selecting a “Field to predict” and “Fields to use for predicting,” we
need to select the Algorithm to be used for prediction.
Splunk’s ML App consists of various algorithms, and each of those has its own importance and
a specific condition where they are best suited to apply.
For our example, we will be using Linear Regression.
Once the “Field to predict” and “Fields to use for predicting” are selected, we
have to split the data into training and testing set.
The reason why we should split our data is that we want to use the training data to create
our model which will be used to predict ‘Remote Access’ and to test the trained model we
would need a dataset which our model has never seen.
Let’s pick a 70:30 split to withhold 30% of the data for testing.
Next, we have the option to save this model as per a naming convention that we can find
on the “Load Existing Settings” tab; we’ll revisit this tab later in this video.
We also have the option to fit intercept in the model's equation or remove it.
To remove intercept, we have to be sure that when all the predictors are having value zero,
then to be predicted value should also be zero.
For example, in the case of mobile logs when the battery percent and the number of apps
open are zero we can say that screen brightness would also be zero.
Therefore, intercept fit can be unselected.
Note: Do not unselect the fit intercept option when you are not sure.
After clicking the fit button, a model is built using the training data.
To evaluate how well the model predicts the test data there are several visualizations
and statistics generated automatically.
The first visualization, “Actual vs. Predicted Line Chart”, compares the ‘Remote Access’
values predicted by the model to the true values in the order they appear in the dataset;
better models produce more similar lines.
Next is the “Actual vs. Predicted Scatter Chart”, which can reveal patterns of misprediction.
In this example, it’s clear that our model estimates Remote Access usage pretty well.
Residuals, or prediction errors, are plotted in the “Residuals Line Chart”.
This corresponds to the difference between the two lines plotted above in the “Actual
vs. Predicted Line Chart”.
Better models produce smaller residuals.
The last visualization is the “Residuals Histogram,” from which it is evident that
there are some values above 100, corresponding to the high-end mispredictions.
Different visualizations can reveal different properties of our model.
The assistant also computes validations statistics.
The R-Squared value represents the strength of the correlation between predicted and actual
values.
It varies between zero and one; closer to one is better.
In this example, the R-square is 0.94 which means we have a good correlation between Remote
Access and our predictors.
On the other hand, Root Mean Squared Error describes the prediction error; closer to
zero is better.
Note that these scores are not always a good rating, for instance, a doctor predicting
if a patient has cancer or not, getting an R-square as 0.94 suggests that out of 100
successful predictions, 6 predictions could be wrong.
That is, some patients suffering from cancer are told that they do not have cancer.
So in this case 0.94 is not a fair score.
The “Fit Model Parameters Summary” provides a summary of the learned model.
In this examples, we’ve used linear regression, so the model consists of the coefficients
and intercept.
(Note that a large coefficient does not imply that a field is important.)
Based on the large deviations between actual and predicted values alerts will be generated.
Using these alerts, network administrator could further investigate for the reason behind
the increase in VPN usage.
This deviation could be due to unauthorized downloads, unknown virtual pc setups, network
breach etc.
On the “Load Existing Settings” tab we can compare models trained with different
parameters using statistics such as R-squared and Root Mean Squared Error, etc. and reload
the one that worked best.
Once we are happy with the validation statistics, we can set the training set to 100% and deploy
our model.
We can also apply our configuration to new data.
These assistants not only walk you through the steps required to train, validate, and
deploy a machine learning model, but all of these tools can be reused and provide a variety
of options for operationalizing our model.
You can view the SPL commands used at every step, schedule the configured setup of the
model, create alerts based on the output generated or use one of the validation panels on your
own dashboard.
To learn more about the Machine Learning Toolkit, including the other Assistants, browse through
our ML videos.