Hi! In this demo, you are going to learn how to perform log analysis using Apache Spark and Oracle NoSQL Database.
For the purpose of this demo, we have already installed Oracle NoSQL database and Apache Spark.
Oracle NoSQL database is running on port 5000, and has a store named KVSTORE.
To perform analysis, We will be using the log files generated from a performance test in one of the NoSQL machines.
Let us open a log file to look at its content.
Each of these log files have 5 fields
LOG_ID, which is the unique ID generated for each log file.
the TIMESTAMP in the form of the number of milliseconds. This allows for filtering of logs based on time periods such as since last hour, day, and week.
LOGLEVEL or the severity of the log, which is INFO, SEVERE or WARNING
the HOST in which the data is stored
the message saved into the database
We will be pushing this data into the NoSQL Database in the form of a table using the Pushlog.java file.
Before we push the log data into the database, let us set the java classpath to specify the location of Spark and NoSQL packages.
We set the classpaths using the setcp.sh file.
This file specifies the location of all the Spark packages and NoSQL packages to the java compiler.
Now, we are ready to push the data into the Oracle NoSQL Database, for this, we use the PushLogs.java file.
Let us open this program in the Vi editor and understand the flow of steps.
This program stores the log files into the NoSQL database and reads data to Apache Spark.
We use the CREATE TABLE statement to store the data from log files.
Log files from all the nodes are collected into a single place before processing and then,
for each file the log entries are parsed into fields and pushed into the Oracle NoSQL Database.
Let us scroll down to the main class.
Here, we create an object called kvconfig and link this object to the store
named kvstore that we have created in the NoSQL Database.
Let us exit from the vi editor and compile this program.
The program is compiled successfully.
Now, let us run the program using the java command and providing the location of the log files
files from where PushLogs takes the data to be inserted into the database.
The screen displays the total number of log entries being processed.
The LogAnalyzer program is used to read log data from the NoSQL Database and perform analysis in Spark.
Let us open this program in Vi editor and understand the flow of the program.
Now that the log entries are in the database, we create a SparkContext object named sparkConf , as highlighted on the screen.
Then, we read the log data from Oracle NoSQL Database into Spark
The classes PrimaryKey, Row, and TableInputFormat are provided by Oracle NoSQL Database.
PrimaryKey and Row are the classes for keys and values respectively, and
TableInputFormat is the class to integrate Spark with Oracle NoSQL database.
It handles the read access from Oracle NoSQL Database.
It reads the configuration parameters such as, kvstore, tableName, and hosts to connect to the database.
A successful execution of this method returns an RDD.
Next, we specify filters to the tool from the command line. If filtering is requested,
the filter method is used on this RDD by passing an appropriate filter function.
The call method takes a row as an argument and returns true if the logLevel matches the required level.
This method gets executed on all the rows and finally the filter method returns a new RDD containing only the required rows.
Step4: Categorizing the logs for easier comprehension
While doing log analysis, you may want to know information such as how many process crashes
or transaction failures occurred, without caring about the process ids etc.
This code allows you to perform simple categorization by replacing the numerical values in the
log messages, and makes the log comprehension easier.
Now, let us compile this program.
The program compiles successfully.
Step 5: Specifying filters to the tool from the command line
Let us run the LogAnalyzer program. This allows you to use filters to the tool and perform analysis in Spark.
Let us search the log data for the word named “time”.
There are 212 log entries containing the word “time” in the message field.
Now, Let us look at all the log data that was created in the past week.
The result shows that there was no log data created for the past week.
Lastly, let us find all the log entries with a WARNING message.
There is 1 entry for this search.
These filters are required to find out how many process crashed or how many transaction failures occurred to perform further analysis.
Note that the LogAnalyzer program is written to perform simple analysis.
In a real application, it is possible for you to do a more complex analysis.
This brings us to the end of this demonstration.
Thank you for watching this demo!
To learn more, visit us online at cloud.oracle.com