hello and welcome to part two in the
watcher lab series in the first video of
the series we introduced watch are the
alerting and notification plug-in for
elastic search we kicked off the series
by looking at the basics of setting up
in a lot we created a simple alert that
queried web logs for certain key phrases
that were indicative of a web attack and
when our alerting condition was met we
logged a text message to elastic search
logs in this video we'll dig a little
deeper into the alerting framework and
leer in some complexity into our
alerting rule will work with server
access and authentication logs in this
example our locks are being indexed into
elastic search in real-time and here we
have them visualized in Habana the logs
show which user attempted to log in at
what time into which machine from which
IP address and then finally whether the
login was successful or not we're not
really seeing any failed login attempts
on this page sure but if I expand on the
field the rough ratio of successful to
fair logins seems to be pretty healthy
we want to use these logs to detect and
alert on brute force login attempts to
gain access to our servers this will be
typically indicated by multiple failed
login attempts typically coming from the
same source IP in a short span of time
so let's be a little bit more specific
with our alerting condition we want to
create an alert when there are more than
10 failed login attempts from the same
source IP in a span of five minutes so
with that goal let's get started we'll
start with a blank watch template a
quick refresher a watch has these five
components trigger input condition
transform and actions to learn more
about the basic anatomy of a watch I
encourage you to go watch the first
video in the series
now let's start filling in the blanks to
set up our alert first we need to
specify the watch execution schedule
we'll use an interval option here to run
this watch every 10 seconds next is the
input block our loading condition
requires the number of failed login
attempts per source IP in the last 5
minutes I'll use the search input type
here to gather that information from our
logs the request is being won against
the authentication logs index next is
the request body to save me some typing
I'll paste this query text the query has
two components the first part is the
search query it finds any documents in
the last 5 minutes where the login
outcome was filled password if you've
seen or use the elasticsearch Gradius
all before you recognize that this is
just a standard elastic search query and
that's again the beauty of watcher if
you can create you can alert on it but
the queries are not limited to just
search we can also bring in elastic
search aggregations once I have all the
documents matching this search query I
want to group them by source IP this
will give me the number of phase logins
on a per IP basis and that's the metric
I need to pass on to my condition block
I'll use terms aggregations here to
calculate the per IP results by default
the aggregation will return the top 10
IP addresses with the most failed login
attempts and I'll just stick with a
default for now so moving on to the
condition block we want to trigger an
alert if there's even a single IP with
more than 10 failed login attempts keep
in mind that our aggregation results are
ordered in a descending manner that is
the first bucket in the result
corresponds to the IP with the highest
field logins so if that first bucket has
more than 10 documents that is if that
first IP has more than 10 failed
attempts we can trigger an action I'll
use the compare condition here and I'm
grabbing the dark count value in the
first bucket of the failed IP
aggregation from my watch payload the
dot 0 here is used to reference the
first element in the aggregation results
array my condition checks if that value
is greater than 10 next up is the setup
for transform I won't be needing the
transform block for this particular
watch so I'll simply delete it finally
moving on to watch action in our last
video our watch action involved logging
a message to elasticsearch logs for this
example we be using the email action
type to send a notification Y email if
our watch condition is met let's copy
the email admin template and fill that
in I want to send the notification to my
email account with this subject line and
I'll leave the rest of the email body
plank now for email notifications to
work we first need to have set up the
email account that watcher can use to
send the notification emails the setup
is pretty straightforward and involves
adding the account details to your
elasticsearch llaman file i've already
configured the email account and this is
what it looks like in the yama file when
you add this be sure to restart
elasticsearch for the changes to take
effect alright so let's go ahead and
register our watch and now to show this
in action I'll run a Python script to
generate a fake attack since this watch
is scheduled to run every 10 seconds we
should see an email pop up
any second now and there you go one last
thing before we wrap up here our watch
is executing every ten seconds but it's
looking for a tax in the last five
minutes
as a result and getting multiple
notifications basically every ten
seconds about the same attack and this
will likely continue for at least in
under five minutes
to address this and to avoid getting
multiple notifications from the same
trigger we can set a throttle period in
our action block I'll set my throttle
period to five minutes and this will
prevent repeated execution of the same
action in a five minute period that's it
in today's edition of watcher labs stay
tuned next week for our next video in
the watcher lab series thank you