>>SHAY BANON: And, at the end of the day, we would like to say it is just another index
in Elasticsearch, and whether that index is logs, or whether that index is a metric or
monitoring index, or whether that index is APM, you can now take all of these three indexes
and make interesting realizations as a result of them.
And, in order to show you just a tip of what's possible if you take all of this data and
put it together -- Logs, metrics, and API --I would like to welcome Tanya on stage.
Tanya?
>>TANYA BRAGIN: Thanks, Shay.
I’ve got a long walk here.
All right.
So this demo scenario should be familiar to many of you just based on your experiences
shopping on the internet.
You can look for an item on a site, you can examine it, decide that you want it, add
it to your cart, and hopefully purchase it.
The infrastructure is underlying the site is also based on fairly common components.
You’ll recognize Django for application servers, Apache Engine X for proxies, MySQL
for database, and of course, Elasticsearch for site search.
From a monitoring perspective, we are doing exactly what Shay just mentioned, we are bringing
together all of these historically disparate sources of operational data into a single
instance of Elasticsearch dedicated to monitoring this site.
So we’re taking the Elastic APM Python agent and that is running within Django, looking
at durations of operations running within the application server and errors.
We are taking Metricbeat and Packetbeat, something that Monica talked about earlier, to look
at infrastructure metrics.
And finally, Filebeat and Auditbeat add log information and other event data to help
us with the root cause analysis of all these KPIs.
We of course have Kibana sitting on top of all of it, and we have machine learning and
alerting running on all of this data holistically to give us a view of all of the anomalies
that we detect on top of what is happening within this site.
As an operator of this site, one of the things I'm really concerned with is abandon rate.
This is when users place something in a cart but then don't purchase it.
Of course, this is going to happen, some users will change their minds, but what you don't
want to see is a sudden spike in abandon rates that’s unusual for the time of day or the
week.
Usually that indicates that a site is not performing well and is preventing users from
buying your products or having a good experience.
So, of course, I'm very concerned when I see on the top of my inbox exactly that: a
high abandon rate anomaly detected by a machine learning job looking at that specific KPI.
Looking at this machine learning job, the silver lining is at least we detected this
right when it started crossing an important anomaly threshold.
So let’s go ahead and troubleshoot this.
This is a Kibana dashboard that starts at the business-level view.
So, I'm using transactional data that’s recorded from the application straight into
Elasticsearch to look at various business-level KPIs, including above-mentioned abandon rate.
And here is a recent spike, and here is this blue line inching toward the top, not something
what you want to be your business trend.
Looking below, I actually notice that my sales are affected, this is not something
that is a red herring, this is something that’s immediately affecting my business and I need
to investigate as soon as possible.
What’s really cool about this dashboard is that in addition to the business KPIs,
we can weave in APM data that is most critical to this view.
In this particular case, I want to see any user-facing errors on the site right next
to all these indicators to see, kind of, the first level of potential root cause.
And in this case indeed there's a cluster of errors happening right now So, what we
will do is jump to investigate to see if that is relevant to what we are looking at.
This is the APM UI that Ron just showed you, and I’m not going to go through this
in detail, but I’ll just look at the errors in particular, and there are a few of them
that are possibly innocuous, but this checkout internal server error looks really bad.
And if you’re a developer you’ll notice this is a number parse exception on the phone
number validation procedure.
Thankfully, this is probably very easy to fix, we just need to do that immediately,
because clearly, this is not allowing users to purchase items from our site.
And now I wish I was done, but nothing in operations is that easy, as you probably know.
If you look back at some of the previous spikes that didn't trigger the machine learning job,
because maybe the average rate wasn't as high, those look pretty concerning as well, and
those happened way before the errors starting clustering together.
So possibly, there's something else happening on a site as well that we should investigate.
As you probably know, another problem that's commonly felt by users on sites, when they
walk away, is slowness -- if the site doesn't load, or partially loads, most of you will
probably just say never mind, I'm going to go somewhere else.
As an operator of the site, I'm again very concerned to see site performance anomalies
found in this data set.
And those go way back so let me start with the one that's the earliest, and again, this
was detected by a machine learning job.
I’m going to go ahead take a look at that.
And the first thing I note, in this view, is that the slowness is not pertaining to
a particular part of the site, this is spread across many important views such as basket
view, order complete view, default check out, all important views but not just one particular
page.
When I see that as an operator, I typically suspect an infrastructure problem.
That's where I will look at first, I will look at my infrastructure analysis dashboard.
Again, this is a Kibana dashboard.
Everything in here is just based on another index in Elasticsearch but It is bringing
all of the data together to troubleshoot this scenario.
We see the site response times from APM, we see things like memory usage, CPU usage, and
total disk I/O from Metricbeat, network data from Packetbeat and finally, slow logs from
MySQL and Auditbeat data.
So, let’s go ahead and take a look first at site responses, that's where we will start
with the user experience.
We see that this started on January 20th, and if I just look at the timeframe when this
began, a couple of things I note.
First of all, this memory profile doesn't look great.
That's very concerning to see the memory consumption of my database goes up on an unbounded manner.
But these CPU spikes also look very strange, very regular, and something that I will just
investigate first, given that they are, you know, very obviously strange and across the
whole infrastructure.
Zooming in on one of those spikes, I can see that there's a lot of traffic going between
the MySQL database underlining the site, and something called proxy_0-1, which I know is
my internal proxy.
I don't expect that to happen, I don’t expect that much traffic happening between
an internal proxy and my database.
So let’s go ahead and take a look just at that proxy.
And a curious thing pops up.
So we see Auditbeat, one of the metrics it looks at is unusual processes, processes that
were started on all of our hosts.
And something called reporter.py jumps to the top of the list.
Let’s take a look.
That sounds like a reporting job.
Again, I don't expect that to be happening.
If I can only click in the right place, we can see that this job, indeed, is happening
very regularly.
If we look closer, we can see that it is happening every minute.
Now, I think that is suspicious but if there's a business need to do that, we need to move
that off of our production database.
That is definitely not something that the site was designed to do, from a load perspective.
We see, if we kind of go back and zoom out, back to a wider time interval that, that,
you know, this kind of load is causing a lot of problems on the database itself, very slow
select statements across the whole site, so we need to isolate it to some other place.
So, in summary, this demo really highlights how elegantly and simply Elastic Stack bringing
together these historically separate sources of data that are important in operations:
APM, infrastructure metrics, and logs.
And, when I speak to users as part of my job, many of you have already embraced that,
and you’ve put logs and metrics into Elasticsearch and you are looking forward to adding APM
to that scenario.
But, for some of you, I find that in your environments, historically, you have relied
on separate tools for these jobs that are hyper-focused on one use case, and that's
nice, but the thing that many of you miss is this ability to bring the data together
to tell a story.
Some of you actually have built custom UIs to do that, that is very expensive, and many
of you cannot really afford to do that.
So, what ends up happening is these troubleshooting scenarios, which should take minutes, or maybe
hours, take days or even weeks, while basically the business is bleeding money.
So my hope is that this demo inspires you to take a look at your operational intelligence,
and think whether Elastic can help you there, and hopefully we can focus on making operations
more efficient while you focus on running your business.
Thank you very much.
(Applause).
>>SHAY BANON: Thank you very much, Tanya.
I love this demo, because it basically proves the fact that every operational problem, it
boils down to data parsing or misconfigured cron job.
(Laughter).