Hello Everyone and Welcome to Splunk Machine Learning Toolkit tour,
Splunk has continuously grown and improved the Splunk Machine Learning Toolkit with a
series of releases since its inception in October 2015.
In this video, we will have a look at the latest updates in the Splunk Machine Learning
Toolkit Version 2.1 and 2.2.
Let’s have a look at the changes in Version 2.1:
There have been two noticeable changes in this version and both of them are accessible
from the “Detect Numeric Outlier Assistant”.
First is the split-by option: Now, we have added the option to specify one or more fields
to split a dataset using the “split-by” feature.
This will enable us in creating visualizations of the dataset which has now been split based
on the “split-by” values.
We can choose upto 5 fields to split a dataset.
Once the split-by fields have been selected, we can see the outlier visualization for each
split-by entity.
Second is the enhanced data distribution histogram which helps us count the no. of points per
standard deviation that lie under and over the threshold limit.
You can get the required SPL to build this visualization from the “Show SPL” option.
Next up are the changes in Version 2.2:
In this version there are two major updates to the Splunk Machine Learning Toolkit,
let’s begin with the changes in the Preprocessing feature.
The Preprocessing feature present in the Cluster Numeric Assistant has now been redesigned
and has been added to the Predict Numeric Fields, Predict Categorical Fields, and Clustering
Numeric Events assistants.
Majority of the times, we cannot use a raw dataset directly for machine learning operations.
Preprocessing allows you to run certain algorithms on your data to prepare it for such machine
learning operations.
For example, if your data contains a large number of fields, this new feature will help
you apply PCA to reduce the no. of fields or if the fields have various scales, then
this feature will help you normalize it.
That was a lot of words!!!
Now Let me show you how it actually works.
You can add one or more preprocessing steps to apply sequential transformations to your
data.
After you apply each preprocessing step, you can review the results to your data to see
if you obtained the desired results.
If you are not satisfied with the results, you can remove the preprocessing step, modify
the settings in the preprocessing step, or add additional preprocessing steps to apply
additional transformations.
Preprocessing steps are saved in the assistant’s history alongwith all the other settings used
in the assistant and can be accessed from the “Load Existing Settings tab”.
Currently, we support three algorithms for preprocessing:
• StandardScaler • PCA
• KernelPCA
This list will grow with future updates.
Continuing with the new features in 2.2 release, we have refactored the process of importing
new algorithms to Splunk Machine Learning Toolkit and we are calling it ML-SPL API.
The Splunk Machine Learning Toolkit natively contains 27 algorithms that you can use to
create machine learning models.
You can extend the Splunk Machine Learning Toolkit with over 300 open source Python algorithms
from the scikit learn, pandas, statsmodel, numpy, and scipy libraries.
These open source algorithms are available to the Splunk Machine Learning Toolkit through
the Python for Scientific Computing add-on.
The new ML-SPL API makes it easier to import an algorithm for developers to work with.
In the Splunk Docs page for ML_SPL API, we can find the section that provides step-wise
guidance to the developers on how to use the ML-SPL API to write custom algorithms.
In this guide, we have 4 different sections:
1) Overview: Which talks about the actual code we have to write, methods we have to
create, arguments we have to pass, classes we have to call etc.
2) Examples: Here you can refer to 4 different examples of importing an algorithm.
This will give a more detailed guide to the developers during implementation as they will
have something to compare their code with.
3) Debugging: One of the most important aspects of development is debugging.
This section is about how to make logging and user-facing errors easier to manage.
This is important because the Splunk Machine Learning Toolkit relies on a different python
interpreter than the one shipped with the Splunk Enterprise.
4) Lastly, we have listed some best practices to follow during the implementation of any
algorithm.
Thank you for reviewing the changes to the Machine Learning Toolkit Version 2.1 and 2.2
with me.
There are a lot features that are still under the development phase and more visualizations,
customization options, algorithms and Assistants are on the way!
Keep updating the toolkit from Splunkbase whenever a new version is posted, and Happy
Splunking!