welcome to the third video when I watch
a lab series in the first video we saw
how to get started with alerting with a
simple search fields trigger the second
video went a little deeper into the
alerting framework and showed you how
you can use elastic aggregations in your
alerting rule both these videos use
static thresholds in the watch condition
in this video we'll look in the dynamic
thresholding where the threshold in the
watch condition is evaluated in a more
dynamic and adaptive way we'll be using
an application metrics data set here
which is tracking the number of active
users logged into our application this
metric is being captured every few
seconds here we are looking at the
average hourly numbers over the last few
days our objective here is to trigger an
alert if we see a sudden drop in the
number of active users you can see from
this chart that the number of users has
a periodic pattern to it the daytime
high is around fifteen or sixteen
hundred users and the data and low is
around six hundred users so setting a
static threshold to detect a sudden drop
or a sudden spike will not work here
there are a number of ways to achieve
dynamic or adaptive thresholding the
overall trend in our example is pretty
gradual and we're mostly interested in
identifying sudden large drops so we'll
go ahead and identify those large drops
by comparing the average in the last
five minutes to the average value for
the five minutes prior to that so with
that alerting plan let's get started
once again I'll be using cents to define
my watch we'll start here with a blank
watch template first we need to define
the watch schedule we choose to run this
watch every 30 seconds using the
interval option now moving on to the
input we'll be using a search input type
and we're running this against the app
metrics index pattern I'm interested
income
preparing the average for the last five
minutes to the average for the five
minutes prior to that so first I'll use
a range filter query to grab logs for
the last ten minutes next I need to
aggregate these results into 5-minute
buckets now there are a number of ways
to achieve that in this example I'll use
a filters aggregation I'm calling it
five minutes here and I have two range
based filters sure both on the timestamp
field
the first filter called latest five
wraps documents in the last five minutes
and the second filter called previous
five grabs documents in the five minutes
prior to that so from now - ten minutes
- now - five minutes so now that I have
my results aggregated into five hundred
buckets I need to calculate the average
value for the number of users in each
bucket so our tag on a metrics
aggregation here called average user
that calculates the average value on the
user's field next we move on to the
condition block I'll be using a script
condition type here keep in mind that
you need to have dynamic scripting in a
board in your elasticsearch llamó file
for this to work for this rip condition
I'm comparing the average value in the
latest five bucket that is the past five
minutes the value in the previous five
bucket that is the five minutes prior to
that my condition will evaluate to true
if there's a greater than to four drop
between the two buckets next is the
transform block which we won't be
needing in this case and finally the
watch action in this example I'll choose
to send a notification wire slack for
slack notifications to work we first
need to configure a slack account in our
elastic so chiama file the setup is free
straightforward it's well documented and
you can check out the watcher
documentation for the instruction
we set up our slack account in the
elasticsearch config in this example
here I'm sending an alert to the watcher
alerts channel and that's my
notification message I want to modify my
notification message to include a value
for the relative decrease in the user
activity so to evaluate that I'll add a
transform block to my actions block here
that will evaluate the full decrease I'm
using a script here that's basically a
simple division that calculates the full
change and the value is stored in the
field called full drop in the payload
I'll modify my slack notification here
to use that field and I'm using double
curly braces here to reference the value
from the payload alright then we have a
watch configured let's go ahead and
register this watch I'm calling it
active users drop to test that this
watch works I've kicked off a script in
the background that will cause a large
drop in the user activity now wait a few
seconds here for the changes to
propagate and trigger an alert and there
you go we have our slack notification
that's it for this edition of watcher
labs
stay tuned for the next video in our
series thank you