>>SHAY BANON:
Another area that is close to our heart, and is being used quite a bit with our stack,
is geo.
So if you think about it, sometimes it is surprising, and sometimes it is not.
But geo is an extremely popular and useful feature in our stack.
If you look at it from the application search perspective, you want to be able to go in
and, for example, filter results, restaurants, based on a bounding box on your iPhone, or
location, or a portion of the map the user is looking for, so it is heavily being used
in search use cases for application filtering based on geo-bounding box or shapes.
Geo is also used for ranking, for example if you want to potentially have the distance
from your location influence the ranking of results that you display to the user.
All of these geo features, as you can imagine, means that we need to invest a lot into Elasticsearch
and Apache Lucene to make it a very strong and capable search engine, which we have done
over the past few years.
We started with BKD trees and tons of investment that we have done in this area.
If you look at it from an analytics perspective, for example, when it comes to logs, if people
are hitting, or accessing, your website, they have an IP address.
You can take that IP address, and then create a geo location look-up on it and then being
able to understand who visits your website from which countries or which location, or
potentially in the security space trying to understand who is trying to hack your website.
This means that you need to be able to take a lot of latitude and longitude points and
be able to go and aggregate them at different levels and be able to display a map of it.
There's another type of geo integration that we started to see happening and being
requested with us, which is basically start with the map, and start to take the data that
exists in Elasticsearch and create layers of investigation on top, or understanding
on top of that.
And we've heard you, and I would like to invite on stage, Thomas, to show us a preview
of an application that we are working on in the GIS space.
Thomas?
(Applause).
>>THOMAS NEIRNYK: Hi, everybody.
I would like to show you a sneak peek of what we are working on for geo right now across
our stack.
What I’ll do here is I will basically give you a big tour of all the features that
we have in the stack right now and how we intend to bring this together into a new UI
experience inside Kibana.
So what we are looking at right here.
This map that you see here is published by the Elastic Maps Service.
The Elastic Maps Service is a central service that we host and that publishes reference
data that you can use.
If you are a user of Kibana, you may have already used it and seen that map right now.
What I want to bring to you soon is the ability to apply styles to this data.
So, for example, we will take the road map layer and then you can apply a stylesheet
to it, for example, a dark theme or a more muted theme.
And the idea, too, is that you would be able to offer your own styles here, too.
So let’s for this demo settle on the dark theme.
Our first layer, our context, is our world map, and I just put on an icon right here
where we are right now, The Masonic, so that’s our second layer, to get a sense of where
we are right now.
And to give you a better sense of this whole layering idea, we will create a couple maps
together right here, all around a central theme.
And the thing we will work on will be the community of Elastic.
So the first map we will create is a map of basically ourworld-wide community, based
on download statistics of our stack.
So I have indexed in Elasticsearch all our downloads from our products: So Elasticsearch,
Kibana, for the last couple months.
And we can look at this as a heatmap.
And this is nice, and it gives us somewhat of an overview of where our downloads are
coming from, but it might not be detailed enough to actually answer very specific questions
of which countries have the most traffic.
And so maybe this is not the right map that we want to create.
What we want to create is to see this by country.
So we will use the Elastic Maps Service to bring in this data.
What I forgot to mention earlier, too, is the Elastic Maps Service does not only serve
a road map, but you can also bring in satellite data, if you have data that would fit that
context.
But, instead, we will actually bring in the world countries, because we want to create
a map where we see, by country, where our downloads are coming from.
So let's bring this in.
And what it is doing right now, we are downloading from the Elastic Maps Service, this layer,
and indexing it into Elasticsearch.
And why we are doing this will be relevant in the second half of this demo.
So here we have, we are looking at the world countries, and now we want to symbolize each
country by the number of downloads it generates.
So we go in the styler, and what we will do to attach to this world layer, you are
going to calculate the number of downloads per country.
And, like Shay said, if you use, for example, Logstash, there's an IP plugin that allows
you to transform an IP to a lat/long location.
That's what we've done and why we have locations, and the other thing it does, too, is attach
a country code to it so we can join it to this layer.
So the world layer has the ISO abbreviation for our country, and we’ve indexed each
document with this same code.
So that allows us to join these two things together.
You could do any aggregation, basically, but let's stick with count.
So let's do this.
And maybe let's look at this already right now.
We immediately see that the United States pops up for the majority of our downloads
was really from the United States, and we have about 14 million in the span of about
three months.
It is not that great of a map, so let's adjust our color ramp a little bit to bring out the
nuances on the bottom end of our downloads data.
So we created a map right now, and actually, this map, the heatmap that you saw, it is
already something that you can do in Kibana right now.
So for the second half, I thought we would create a different map, but illustrate a new
functionality that we are working on.
What we want to bring into Kibana is full support for the geo-shape datatype in Elasticsearch,
and that gives us more power to work with and will allow us to do more advanced functionality.
One of the things that we will be able to do is visualize individual documents, and
you can already see it in the world map.
So each country is a document, and the other thing we will be able to do is do these kind
of analytics on the fly.
So let’s do that right now.
The second map you will make is related, but I thought we would make a map of where we
all come from.
So I got a data layer, basically the location from all of us where we work.
This is anonymous data, don't worry.
And there's about 2,500 of us, right here.
So this is actually custom data, and what I did, I imported the GeoJSON file, and the
nice thing about GeoJSON is that all of the types of geometries you can support in it
are also supported by Elasticsearch, so the indexing step is fairly straightforward, so
we’ll bring this in, and again this is now indexing into Elasticsearch.
Here is the 2,500 of us, but we have a very similar problem that we had with the download
statistics, is that it gives us an overview, but it does not actually say anything specific.
So what we will do now is we will make another world map, so let’s deactivate this layer,
but symbolize it based on the count of attendees here at Elastic{ON}.
So we take this layer, and we are going to count, for each country, the number of attendees.
What we used in this count actually uses the spatial filtering functionality already in
Elasticsearch.
It is very nice to be able to do these king of aggregations on the fly.
So if you look at it, and actually, we will hide this for a minute, we have a very similar
map, as earlier, that the majority of attendees are from the United States.
But, instead of just, you know, playing with the color ramp, because we can do this on
the fly, this also means we can refine on the fly.
So let’s actually bucket our data not only by world country, but also by United
States’ state.
So for our reference data, we will again go to the Elastic Maps Service because it
has reference layers that we can use.
We will bring it in, indexing into Elasticsearch.
And we will install the layer the exact same way that we did earlier.
So to each state, you are going to attach a count of the number of attendees here at
Elastic{ON}, which you do right here.
We see that California jumps out, about 450 people are here from California, which
makes sense, because it is local.
We have the same problem, again, because the nuances at the bottom don't come out.
So let’s adjust our color ramp once more.
Our maps are starting to get to look like the way we want it, to give a global overview
of where we’re from, and get a sense of which states or which countries sent the most
people here to Elastic{ON}.
Because this is layered and we have five layers on the map right now, it also means that we
can sort of individually rearrange and position them as we want.
So I think this map would make more sense if we combined the states and the world countries,
and maybe move the venue location to the top too, so it pops out more.
And, you know, just to get some more detail we’ll back the attendees, but we’ll star
them a little smaller so they don’t dominate the map as much.
The language I am using here is the Mapbox CSS language, a styling language to apply
to vector data, like we do here.
So we will apply it.
This is how we created our map of all of us right here, and that was it.