machine learning has a little-known
feature called categorization let's
learn how it works
categorization is automatically grouping
similar log messages together into the
same kind of category it's done via an
algorithm that looks at string
similarity and cluster similar log
messages together you'd want to do this
because we need to bring structure to
otherwise semi structured data so that
it can be analyzed and the real benefit
here is that we can find anomalies in
log files without knowing exactly what
they contain
let's look at an example here I have a
log file from a Linux system how would
categorization work on this well the
first step is that machine learning
would take and remove what I'll call
mutable text from the messages this is
text that's likely to change IP
addresses timestamps ultimately what we
want to do is we want to filter those
out and then move on to step two which
is to cluster those similar messages
together so if we look at this we can
see that there are a variety of
different categories that can exist here
some of which have just maybe one log
line that's in that category or others
like category 6 can have several
examples in there where the wording is
just slightly different so now since we
have those categories identified what we
can do is we can count them for every
time interval so for example in the
first time bucket we may see one
occurrence each of the first three
category types similarly in the second
bucket we would see one occurrence of
category 4 5 and 6 and then in the third
bucket perhaps we see four occurrences
of category 6 and so on so we're just
tracking the occurrence rate of these
different categories over time and of
course we want to do anomaly detection
if there is an unexpected spike of a
certain type of category so let's see a
little bit of a demo inside of kibana
here I can see a log file that's been
ingested it has a timestamp in a message
field and again the message field is
pretty much arbitrary text when we go to
machine learning what we need to do when
we create a new job
is to point at that particular index or
that data exists go to the advanced job
option and in here after we give the job
a name we're going to go over to the
analysis configuration and select which
field of the document is going to be the
field that we actually want machine
learning to categorize so if there
happened to be more than one field we
would pick that in this case there is
only one field of interest and that's
the message field
once that's done when we go to the
detector we add a new detector here we
can just select the analysis that we
want in this case I'm going to do a
count of messages by ml category now
this field ml category gets magically
created once categorization is signified
so again you would not normally see this
field if you weren't doing
categorization but because we put the
categorization field name in then this
ml category field now shows up okay so
let's just add that in save the job and
when we start the data feed and let it
run through what will happen is machine
learning
well again do all those things that we
talked about in the presentation and
when I click on the result
I'll see what I get here for the
analysis and we found a few anomalies
here in this case this is for our ml
category number 20 and there was some
potential here maximum authentication
attempts let's look at some other ones
here get too many authentication
failures and so on you see these are all
different category types and again the
idea here is that without really knowing
what to look for in the log messages
machine learning has automatically found
very very interesting authentication
anomalies here so
again this is something that is
relatively easy and straightforward to
take advantage of if you have
semi-structured log lines that you don't
really know what to look for in thanks
for watching we'll see you next time