The best and easiest way to analyze data in Elasticsearch is through the rich
user interface of Kibana. In this video, we will show you how to get started with Kibana.
Here we have a sample dashboard. It is based on public data from the New
York City Taxi and Limousine Commission and covers almost fourteen million trips
from June 2015. Kibana dashboards makes it really easy
to explore your data, drill down, and follow your hunches. Here we are seeing
the daily number of trips from June of 2015.
If we wanted to look closely at a particular week or day, I can easily zoom
in on the time period in the visualization. Notice how the rest of the
dashboard updates to reflect data from the selected time period. If you're
looking for finer control on the time frame, the time picker is your friend.
Let's reset the time period to the full range - June 2015.
This dashboard is made up of nine visualization panels, each highlighting a
different aspect of our data set. Filters are a great tool to drill down and focus
on a specific subset of the data. For example, let's say we were only
interested in fares paid for in cash. the easiest way to apply a filter is to
click on the filter term on a visualization. Here we filter on cash in
the pie chart. This adds a filter at the top. If we instead wanted to see all
trips not paid for in cash, we can quickly invert the filter. You can see
that it now turns red to show it's inverted. We can also directly add a
filter query. For example, if you only wanted to see trips with fares below $5.
Surprisingly, a significant portion of sub five-dollar rides are paid by credit
card. Geographic filtering is also easy and one of my favorite features of
Kibana. Just select an area on the map. Here we can see the trips matching the
earlier filters that originated near central Manhattan. We can now delete all
filters by clicking on actions and then select remove.
That was a quick introduction into how to work with Kibana dashboards.
Some of you may think that is great but how did we get there? Well, that is what we're
going to focus on next. For simplicity, we will use the same data set we used
earlier. This demo assumes that you already have data indexed in
Elasticsearch. If this is not yet the case, there is nothing to worry about as
we will point to some excellent resources in the video description.
Before you can do almost anything in Kibana, you need to tell it which
Elasticsearch indices to use. We do this by defining an index pattern under the
management app. We know that the indices we want to analyze starts with
nyc_tripdata. We therefore create an index pattern that starts with this
prefix and ends with a wildcard. Next, we will define the primary time field.
This will be used in the discover app as we will see in a minute. We can
now see all the fields in this index pattern and how they are mapped. If
you're not sure about the name of the indices to use, there are a couple of
ways to find out. If you have X-Pack installed, you can see the indices in
your cluster under the monitoring application. We can see here that we have
two indices: nyc_tripdata-green and nyc_tripdata-yellow. These contain data
related to different types of taxes but both matched index pattern we just
defined. X-Pack provides a lot of very useful
features in addition to an easy way to explore indices so I encourage you to
install and explore it. Another way to get information about indices is to go
to console under dev tools and run a query against the cat indices API.
Console is a great interface to run Elasticsearch queries and practice your
query DSL skills. To inspect the data we are working with, we go to the discover
application. If you're not able to see any data, we can use a time picker to
select the correct time frame. This histogram shows the count of distribution
of events over time. To the left is the index pattern and a list of fields. We
can expand these to see the most common values. To
inspect the events we can click on one of them. This gives us a table view of
the data, but we can also see the document in JSON. If we were interested
in creating a visualization based on only a subset of data, we could add
filters here and store this as a saved search but that is not required for our
dashboard. To create visualizations that we can put on our dashboard, we go to the
visualizer application. To create a new one, we click on the plus icon. Here we
can see the full range of different visualizations we have at our disposal
in Kibana. We first want to create a heat map showing trip count per hour and day
of the week and select the heat map icon. When creating a visualization, we can
choose to base this either on an unfiltered index pattern or a saved
search. As we want to base it on all data, we select our index pattern. To the left
we have the panel we use to configure our visualization and to the right we
can observe the result. The default first metric is count which we will keep. We
will however give it a new custom label. On the x-axis, we select histogram
aggregation based on pickup hour of day with an interval of 1 and label this
hour of day. On the y-axis, we also select the histogram aggregation but base this
on the day of week field with an interval of 1 and label this day a week.
Let's see how this looks. This looks good but could use additional intervals. Go to
the options tab and increase the number of colors to 10. Let's save this as
NYC_Hour_day_trip_count. Next, we want to create a data table showing the trip
count and average fare for trips of different lanes. We create a new data
table visualization based on our index pattern. We keep the first metric but
label each trip count. We then add another metric based on an average
aggregation on the fare amount field. We label this average fare. Next, we want to
split the rows based on the trip length. Under buckets, we select split rows and select
the range aggregation based on the trip distance field. We create the desired
range bands and refresh. We then save this as NYC Fare range distribution.
The next step is to create a new dashboard with the visualizations we
have created so far. Click on the dashboard application and create a new
dashboard. Click on add to select the available visualizations. This places
them on the dashboard. Minimize the import panel. We are now in edit mode and
can resize and move panels around until we have them where we want. We then save
this dashboard as NYC Taxi analysis. We could now go through the same process
and add the remaining visualizations. This would however take some time and
might be somewhat repetitive. To speed this up, I have some stored
visualizations that I have prepared in advance that we can simply import. We can
add these through the dashboard and arrange it the way we want.
I often import and position a few at a time but that is just a personal
preference. We will show the first pair in detail and then speed up by just
showing the result of each subsequent step until we get to the final product.
As you can see, getting started with Kibana is easy. In this introduction, we
have however barely scratched the surface of all Kibana has to offer.
Try it out for yourself!