Splunk has hundreds of apps that extend its core capabilities.
The “Machine Learning Toolkit” app delivers new ML Specific search commands, visualizations,
and other tools for performing machine learning on your Splunk data.
Splunk has hundreds of apps that extend its core capabilities. The “Machine Learning
Toolkit” app delivers new ML Specific search commands, visualizations, and other tools
for performing machine learning on your Splunk data. In addition, the application has ML
Assistants that guide you through the process of performing particular analytics, building
custom models, and operationalizing them on the Splunk Platform.
In this video, we’ll explore the “Cluster Numeric Events” assistant, which helps to
group the events based on similarity of numeric fields. This task is commonly referred to
as clustering. There are countless uses for clustering, from identifying the hot zones
in any spread of points to market segmentation. We will cover those in subsequent videos.
Note that most of the input and output panels in this assistant have tooltips, giving us
more information about the particular panel. Hover over a panel title or field label to
reveal the tooltip
For this example, we want to group the hard drives based on their numeric fields. These
groups could signify the performance or health of the hard drive. Using the search field,
we first load the disk failure dataset that is packaged with the app. After clicking the
search button, we get the “Raw Data Preview” panel below, which shows the fields in the
dataset.
We also have the option to preprocess our data before clustering. We can standardize
our fields so that scale differences such as the length of the numeric data in Smart
1 and Smart 4 field are removed and do not bias the clustering. After this, we can apply
Principal components analysis or PCA option to reduce the dimensionality of the problem.
PCA is useful to reduce the no. of fields to be used for processing while preserving
their importance and variability. We will reduce our five fields to three fields.
Next step is to select the algorithm to be used for clustering. Splunk’s ML App includes
several clustering algorithms, each of which is better suited for different kinds of data.
You should experiment with the options to see which one works best for your data. For
this example, we will use K-Means.
In this example, we group the hard drives based on the 3 fields generated by PCA (i.e.,
the principle components). Selecting fields, in general, is a hard problem that depends
on the use case, characteristics of the data, and properties of the algorithm, itself, that
are beyond the scope of this video.
KMeans requires that we specify k: the number of clusters. In this example, we’ll partition
the hard drives into three clusters.
Next, we have the option to save this model as per a naming convention that we can find
on the “Load Existing Settings” tab; we’ll revisit this tab later in this video.
After clicking the cluster button, the hard drives are divided into three groups based
on the selected fields.
To evaluate the clusters and their relation to the fields used for clustering, a Cluster
Visualization is created. This Visualization is the scatter plot of fields selected for
clustering with highlighted clusters in each plot. Hover over any point in visualization
to know its value, and you can also select a group of points in the visualization and
see the corresponding points in the other plots.
On the “Load Existing Settings” tab we can compare models trained with different
parameters using statistics such as Field to use for clustering, Clustering Algorithm,
Dimensionality Reduction, etc. and reload the one that worked best. We can also apply
our configuration to new data. These assistants not only walk you through the steps required
for clustering, but all of these tools can be reused and provide a variety of options
for operationalizing. You can view the SPL commands used at every step, schedule the
configured setup of the cluster, create alerts based on the output generated or use one of
the validation panels on your own dashboard.
To learn more about the Machine Learning Toolkit, including the other Assistants, browse through
our ML videos.