>>SHAY BANON: So rollups is a very interesting feature.
I admit, I had a lot of problems with the notion of rollups, historically.
If you think about a search engine, the power of a search engine is the ability to take
hundreds, a hundred attributes for a specific data point, and being able to slice and dice
your data however you want to see fit.
And whether that data comes from yesterday, or whether that data comes from ten years
ago, that freedom, and that ability to slice and dice your data, is very empowering.
Now, if you look at rollups, and why do people want to roll up data, they want to save on
space.
They want to be able to go and save on space when it comes to ten-year-old data, versus
data from yesterday.
The challenge with rollups is you predefine how you want to look at your data.
And what happens, if you roll up your data from ten years ago, and suddenly you realize
there's one dimension that you want to look at in your data, but you can no longer do
it because you have just rolled up your data.
And so there's value in rollups, and there are challenges with rollups.
So over the past few years, what we've done is focused on trying to make our system best
when it comes to not necessarily needing to use rollups, whether through compression,
whether its being more efficient in storing data, or whether it’s through being more
efficient when it comes to querying and analyzing large amounts of data.
But this year, we felt like it is time to focus on rollups.
And I am super excited, because I’ve actually stepped into this project with our team, and
I had my concerns around rollups which I mentioned before.
And it is very impressive how we looked at rollups, like everything we do in Elastic,
we put our spin into it.
And I would like to welcome Zach on stage to show the roll up feature in Elasticsearch.
Zach?
(Applause).
>>ZACH TONG: Hey, everyone.
So, to show you how the rollup feature works, we’re going to do some good old-fashioned
curl demoing here.
We have a Metricbeat index on my laptop it’s got about 9 million documents, roughly 2GB
in size, and it is Metricbeat data that you would expect.
You have modules that are enabled, each module emits several documents per seconds,
and it is filled with a bunch of numeric values that you’d like to keep around.
The goal of the rollup is to take all this Metricbeat data and compress it to something
much, much smaller, but still make it useful in the future.
I’m going to go ahead and run this so it will process it in the background while
I talk, and let’s look at how how rollups actually work.
This rollup job configuration right here is how you set up the rollup in the system.
And basically, a rollup job is a process that runs in the background and continually
looks over your data, and aggregates it into summary documents.
You define an index, or an index pattern you would like to roll up, so in this case,
metricbeat-*, and you tell it, I would like to stick all of those rolled up documents
into the rolled index.
You give it some type of cron schedule so it runs every hour, or maybe at midnight,
and the process runs in the background and churns through your data.
The rest of the configuration is basically what you would like to roll up.
So there are two sections here, there's the groups and the metrics.
The metrics are pretty straightforward.
These are numeric values inside the documents that you would like to collect — the min,
max, or average of.
And these are really cheap to save, so we usually just save all the numerics we can
find in all your documents.
And then if you go back up to the groups section, the groups are essentially what fields
you would like to run future aggregations on.
What is important about the rollup system in our featture as opposed to other systems
is we don't want you to define specific charts that you are interested in.
Other systems, you have to say, I would like the average CPU on an hourly basis, and
then you get a chart for it.
But what if you need average CPU partitioned by a host name on an hourly basis, or a daily
basis, or any of the other combinations?
You have to define them explicitly up front, otherwise you’ve lost your data and you
can’t really use it.
So we took a different approach where we say, don’t define charts, just define the
fields you are interested in and the aggregations you would potentially like to run searches
on at a later date.
So here we’ve got a timestamp field that we would run a date histogram on at some point
using hourly interviews, and then a whole bunch of numeric and keyword fields that we’d
like to run terms aggregations on.
This is stuff like the process name or what metric beat module it came out of.
Alright, so that job should be done now, that we started.
Yup, that’s finished.
Let’s take a look at what it actually.Let's refresh that.
So you can see the original index, we still have that, it is 9 million documents, but
there's a new rolled index, which has about 50,000 documents, and 27.30MB in size, which
is considerably smaller than what we started with.
And now, of course, it is easy to get good compression if you delete data.
We need to be able to use -- (laughter) -- this very small index for something.
And so, if you look here, we've got a query that looks just like a normal Elasticsearch
query.
We have a term query at the top, there's a date histogram aggregation, there's an average
metric aggregation, and this looks exactly how you would expect in a normal search.
The only difference is it’s going to the rolled index, and the _rollupsearch endpoint,
a slightly different endpoint that you use, but the same syntax otherwise.
We will execute that, and you can see that it gives a response which is basically what
you would expect — buckets, inside the buckets there are keys, and inside each bucket are
the metrics you asked for.
This is important for the rollup future, we wanted the request and the response syntax
to be basically identical to a normal search, so that it’s as close to a drop-in replacement
as possible for your dashboards.
Now of course, you can take my word for it that this is actually real data, but we
can check it if we pull up a visual diff, so we can paste the response there, and go
and execute a search against the raw data.
So here we’ve got the metricbeat-* index, and the regular search endpoint with the same
query, we will execute that and take that response.
You can see that the responses are, in fact, identical.
So even though we’re using this much, much smaller index, we can use the exact same
query and get the same results.
Now, I don't really have time to show you the cool things we can do with rollups, but
suffice to say there's a lot more flexibility here than what you would normally expect from
a rollup system.
So this much more complicated aggregation here, we have a date histogram on daily intervals
instead of the hourly interval before, we’ve got two different terms aggregations that
are embedded inside each other, and then three different metrics at the bottom.
And when you execute it, it runs exactly as you would expect from a regular search.
And what is interesting is this is that it’s all being generated from this small, rolled
index.
And a nice feature of this, is now you have the rolled up representation which you can
use to accelerate dashboards, and you still have raw data you can keep around for more
fine grain analysis.
And at some point, if you don't need your fine grain information anymore, you can go
ahead and deprecate out that data.
So that’s the rollup feature, if you have any questions or this interests you, I will
be at the AMA desk pretty much all conference, so come find me.
Thanks.
>>SHAY BANON: Thank you, Zach.
(Applause)