organizations are storing massive
amounts of production Leda and
elasticsearch the production data
especially with data streaming in
real-time users often want to get
notified when something changes in their
data according to predefined conditions
the application performance monitoring
as an example we are collecting
performance metrics such as response
times and lead and sees that a surrogate
measures for user experience in this
case you want to get notified when these
performance metrics drop below quality
thresholds another common use case is
security analytics where you want to
notify a security analyst when a
security incident is detected in your
logs one such example is seeing multiple
failed login attempts from the same ip
in a short span of time these are just a
couple of examples with real-time
alerting matters watcher which is the
alerting and notification feature for
the elastic stack
enable feeds use cases and many more in
this introductory video series which we
are calling watch a lab will show you
watch our inaction using a variety of
simple examples we'll start simple in
this first video and touch on just three
things
installing watch or creating our first
watch and I will finally wrap up with
the watch or execute API which is a
nifty testing and debugging tool so with
that let's get started we'll install
watcher into our existing elasticsearch
cluster this involves two steps first we
need to install the license plugin and
second we need to install the watcher
plugin
we do both by running the bim plug-in
install command in our elasticsearch
install directory in this video series
we'll be working with elasticsearch
version 2.3 now I have successfully
installed watch
let's restart elasticsearch and hop over
the cabana in this first video we'll be
working with Apache access logs which
are streaming in elasticsearch cluster
in real-time here we have them visualize
in cabana each log has information like
the requested URL at what time the
request was received the source IP the
location of the requester and much more
a goal here is to detect and log
attempts to access the Etsy password
page on our servers such web attacks can
be detected by inspecting the requested
URL in the web access logs running a
simple text query in Cabana shows me
that there were quite a few attempts in
the last one hour so let's get started
with creating this alert watcher
provides a restful api to create and
manage your alerts in this video we'll
be using sense to interact with the
watcher API
since the Cabana plugin that provides a
console for interacting with
elasticsearch api's to set up a new
alert we need to create a watch a watch
is essentially a single alerting rule
and live composed of five parts shown
here in the blank watch template the
five components of a watch are trigger
input condition transform and actions
the first component of a watch is the
trigger this defines the watch execution
schedule what time or how often do I
want to run this watch here I'll choose
to run this watch every one hour using
the interval option watcher provides
many other options to define your
execution schedule and we'll dive into
some of those options in later videos
next we need to define the watch input
the beauty of watcher is that if you can
search for it you can alert on it I want
to
my logs generated in the last one hour
that had the text Etsy password in their
request field when this watched executed
any documents matching this query will
get loaded into the watch payload I'm
running this query against the Apache
access locks index I'm using a compound
boolean query here to look for any
documents in the last one hour which had
the text Etsy password in the request
field if you have use elastic search
query API in the past you recognize that
everything within this query block it's
just a regular elastic search query the
next part of the watch definition is the
condition the watch execution engine
were test the watch payload against the
condition to decide whether an action is
needed or not in this case our condition
is rather simple we want to alert if
there's even a single attempt to access
the Etsy password file in other words we
want to alert if the number of documents
or hints in our watch payload is greater
than zero I'll use the compare condition
block to achieve this the next part of
the watch definition is transformed this
is an optional piece that allows us to
transform the payload prior to taking an
action we'll learn more about this in
future videos I don't need the transform
block for this particular watch so I'll
simply delete it finally moving on to
action we need to specify what needs to
be done if the watch condition is met
you can do many things you are trigger
an email push notifications to slack or
hit chart or pager duty or simply index
an event back to elastic search in this
first video I'll keep things light and
simply log a text message to my elastic
search logs that's it a watch definition
is complete we are ready to register
our first watch using the foot watch API
will call this watch password file alert
now because I've specified my trigger to
run this watch every one hour the first
watch will not execute until an hour
from now but I don't want to wait an
hour to see if this watch works is
expected and the execute API can help
with that testing the execute API allows
you to execute a watch outside of its
normal trigger schedule it's a great way
to test and debug your watches now that
I've executed my watch
let's go to our terminal running
elasticsearch and see if our watch
generated a log you can see here that
our watch worked as expected and an
alert message was loved to elasticsearch
a current watch message only mentioned
that an access attempt was made but does
not indicate how many such attempts were
made in the past hour let's go back the
sense and tweak the watch action to
include that number I'm using double
curly braces here to pass the total hits
value for my watch payload to my message
text let's run this again to update the
watch the put action is acknowledged and
you can see here that the version ID was
updated now let us test the update by re
executing the watch using the execute
API and if we go back to the terminal we
see our modified log message which
includes the number of attempts that's
it in this first edition of watch
elapsed
next time we'll dive a little deeper and
see how to bring in analytics and
aggregations to create more complex
alerting rules stadium in the coming
weeks for our next video thank you