Hunk: Splunk Analytic for Hadoop
Hunk: Index, Analyze and Visualize Hadoop data
On the fly. Hunk Virtual indexes
allows Splunk to index and process Hadoop data
in each data. In addition Splunk
automatically create schema on the fly and there is no need
for () schema the end result is
fast implementation and high degree of flexibility
for Hadoop projects. During the run time:
Hunk would copy the search from the Splunk
to Hadoop it will then transform that
Splunk search into Hadoop streaming
Map Jobs after each hadoop map phase
Hunk would copy these intermediate results into
an HDFS directory and will process these results
in place. At that point the Splunk search aid
will pull the process results
and will display them in the skunk you like in essence
the Splunk search aid act is the
reduce fay's offered and
as we expect once all the Duke map jobs
I've done the Splunk you why will visualize
the full results for
Dez hug him or we're going to see how honk
index analyze and visualize to Hadoop data sources
the first day two souls in a stored in
a direct arena Duke director called user unknown National Energy
it's a fairly large XML file the second one is the
little bit smaller any stored in the user one on
daily energy directory the
just ate to see an example of what the National Energy XML file looks like
it basically is a year XML that is broken by
those it will give us the information
that was gathered in 2005 about the
United States aid energy consumption to set up
virtual indexes DD to Duke directory that we have for this demo
you can go into System in and the
the data section you will see a virtual indexes
link clicking on the virtual indexes link
you see two subtypes the first one is the providers
the second one is the butcher indexes the providers
give us the ability to configure the connection
in the basic setup for the Hadoop cluster in the virtual indexes allows us
to point to a specific directory
that we won't to index looking at
an example that are created for this demo them I had to provide a
we see the basic configuration for Java
for Hadoop the location of the job truck /a
a and the name node in addition to that
we can also see the Splunk settings the stock settings
are requirement since Mike is going to be installed
each data load so for example we need the stunt directory on the day to know
the location for that directory
d XD face working directory is the location where
after each map phase the Splunk
will pull the data from this particular directory by the search ed
as well as the Splunk software itself it going to be stalled in it
be above directory a father more UK knit
tell Splunk to automatically set up its
softer on all the data nodes all you can do it yourself
in there was a few of the settings for example
the Splunk Mr that
job far is the actual file that will translate the Splunk surge
into Hadoop streaming MapReduce is well as the ability to set up a garbage
collection
in many other I'll elements all of these
environment you can add
D virtual indexes supped up
we could see an example all for the Hadoop on the school national energy
index indeed when we set it up
we see that we have to point to the Hadoop directory
where a a the data resides
is well as the type of Oz we want to pull from that
in there we can set up additional customize it
type format timestamp format that we can set up so that when smoke was these data
it gives us a some sort of our a timestamp if you wish to do that
a one more element that we can say it
can be done he a file called
eat rocks that Khan in this case seems to have the
Davey energy AD National Energy
I set up additional parameters for each one of those for example in the national
energies a Hadoop directory I
told Splunk at the skis in XML file
in to break the XML after each row
is well as to set up the Q value
mode to an XML in so that whenever we displayed in Splunk
we have that all setup 4 a.m.
the indexing and a analyzing and visualize
once the bow to indexes have been created all we have to do
is to go to the Splunk search command
and call their
index is that we created a minute ago Splunk
then we'll go to the Hadoop directories and bring just the result
phone that particular directory equal you pics
all data it will break all of the key value pairs
and put it on the left side in allow this to go
a a create individualization
on the fly from that take you by the pairs as well as
we can go into Sloan in tie
be in the spline commands for example stacked
com by Tom a
role a age all the
air conditioning my house I can go we invoke that surge
slut will go in and
had that additional information to the surge
its true to result to Splunk and we can see as
loan goes into ado it goes the NSW that was all
as they are produced by be map produced by the Hadoop MapReduce
we can see the result in a tabular format all weekend CPB civilization get
updated
by Splunk is the event's against him
%um from her do
additionally we can see the actual
details of the MapReduce this was running on the
dupe site so looking at the search
that they created for this them ok oldie national average
with kilowatts four square feet and we to 18
the Splunk smoked mode
I'll a soon as the job
started running it's good to as a mission on its going to
convert the search the skunk search into Hadoop MapReduce job
therefore looking and it details
in Hadoop weekend see the actual
in map in reduce that we've generated
based hard these Splunk search
so gonna give it a few seconds to to Ronnie to
get the result stream into Splunk
and is the job he's running
weekend Luke at two things
first 20 we can look it the Hadoop user interface
a and we can see the progress of the job
he the job trucco user interface their
dump truck to use interface indeed the
job has the specific 8d that was generated
so as soon as its done we can be fresh Dez
user interface in the see the result of the job that we just
executed looking at this particular job
we can see that Splunk actually
generated two maps for as we mentioned earlier Splunk
ease the we do stays you c0
be zoo 0 reduce a.m. JV ends
running also you can see other elements of this particular job
in a few autographs that to you the job trucker generated
if %ah the mall if we look an act the Splunk
you I itself underneath the job
and we can inspect the job itself we
if we go all the way to the bottom of Dez
a.m. display we have to search log which will give us
the actual details of the job
me 8d debt Splunk generated
to get additional information about hunk go to spend that come
and underneath the solutions area you see a link to the hunk
beta click on that it will take you to the page where
you gain additional information about the product
is well as the ability to sign a full be bade ache so
a father more if you go to blogs that spark that come
and in the search area type: Hadoop
you will see link to several additional
a new blogs about the subject
to summarize hunk is a new part of Li from smoke
it allows the exploration analytics and visualization of Hadoop data
without the need for that data to
leave the Hadoop ecosystem order processing
and all the indexing will be done without single
line of code from the user's Blanc will translate
the Query Language to he heard of MapReduce
and the users can benefit from be
Splunk is abuse and the Hadoop
processing power thank you