Hello and Welcome to the School of Splunk,
Today, we are going to talk about What’s New in the Splunk Machine Learning Toolkit
Version 3.1
There are 3 important features which were released in the MLTK 3.1 version:
1) Field Selector Option 2) X-Means Algorithm
3) Categorical Encoding Limit
Now let’s jump to the new pre-processing option: Field Selector, which is available
in MLTK 3.1 and above.
Field Selector helps in selecting the best fields from all the fields for your target
metric’s prediction.
Also, it is useful when you have no prior knowledge about the relation between the metrics
in your data.
Let’s see how it works.
For this example, I am trying to limit the no. of fields used for prediction to save
time and my processing power.
I have selected ac power as target field, and now I am going to run field selector across
all my fields/metrics in my dataset to find the relevant ones (with default settings).
Aha!
Success, we got one field which could be used all alone for my prediction but before I do
that let’s see what’s the model validation stats right now and we will compare it with
the result from the selected fields.
Now let’s run it.
As you can see results are pretty close, even though we used less no. of fields.
And that’s how you utilize the field selector option.
The second feature in our list is Xmean algorithm:
● X-means algorithm should be used when you have unlabeled data and no prior knowledge
of the total number of labels into which the data could be divided.
● The X-means clustering algorithm is an extended version of K-means which tries to
automatically determine the number of clusters based on Bayesian Information Criterion (BIC)
scores.
Let’s understand its usage using an example:
We have uploaded a soybean.csv dataset into Splunk and it looks like this.
In this dataset, we will try to classify different types of diseases a soybean plant can get
which is the field “class.”
There are 19 different types of diseases a soybean plant can get, now let’s assume
we didn’t have this information, and so we are going to use other meta data or metrics
in the dataset to find out how many different types of diseases are possible.
Here is the SPL which we are going to run:
First, we are loading the data onto search; then we are renaming all the fields with pre-fix
feature.
Next, change the name of our class field to target_class because we don’t want to use
this field for our classification.
Now pre-processing our data with Standard Scaler and Field Selector algorithms.
Finally fitting the X means algorithm and then calculating the no. of disease (class)
cluster formed by the X means algorithm.
Walla, we are pretty close to the actual no. of disease classes.
Another use case for this algorithm would be, finding out the no. of jobs running in
a cluster of the distributed system by classifying individual systems and its metadata.
As pointed out earlier, X-Means algorithm is similar to K-Means algorithm but has these
advantages in comparison:
Lastly, we have the Categorical Encoding Limit feature; this is more of a configuration option
rather than a feature.
Previously in MLTK, if we had loaded a categorical field/metric with more than 100 distinct values,
than it gets ignored because of the performance constraints.
Now, we have given the user an option to increase the categorical limit beyond 100 based on
their systems capability.
You can make that change in the “mlspl.conf” file as shown in this page.
This concludes our introduction to "What’s New in MLTK" video and we hope you found our
features interesting and useful.
If you have any questions, then please reach out to us on this mail id.
Thank You and Happy Splunking!