Hello world welcome to Sirajology!
Today we're going to be building sentiment analysis in four minutes let's get started.
sentiment analysis is the process of determining the opinion or feeling of a
piece of text we humans are pretty good at this
I can look at this tweet and immediately know that it's negative it feels like
the writer sentiment is one of Tanger and discussed due to the negative
wording companies across the world have implemented machine learning to do this
automatically it's super useful for gaining insight into customer opinions
once you understand how the customer feels after analyzing their comments or
reviews you can identify what they liked and disliked and built things like
recommendation systems or more targeted marketing campaigns for them in this
demo we gonna be building a sentiment analysis program in Python that will
identify whether a movie review is positive or negative based on the text
in the review will get our training and testing data a bunch of labeled reviews
from a site called Kaggle will certify importing are dependencies to import the
operating system module to help us reform command-line functions then want
to import the SciKit learn module which is a machine learning library in Python
with fast learning curve then We'll import a helper class that will help us
clean our data pandas helps us read our data CSV files and NLT k will be used to
remove unnecessary words from our dataset alright so step one is just read
the data from hard disk will import the labeled training data and the testing
data then we'll print out the first review to the command line to ensure we
read the data set correctly once weve read in our data step two is to clean it that
means ensure that we remove all the HTML non letters and stop words stop words are
words that are insignificant we can download them from the NL TK or natural
language toolkit library words like
or two or as since it's hard to analyze emotion from them over every review in
our training dataset into our new cleaned review array with the cleaned
reviews for helper class will do the cleaning for us
step three is to create a bag of words the bag of words model is a simple
numeric representation of a piece of text that is easy to classify
we just count the frequency of each word in a piece of text and create a
dictionary of them this is called tokenization in natural language
processing will use a count vectorizer object in the scikit learn package to
create it will set the max features to 5,000 to keep things simple to our bag
of words will contain at max 5000 words and their associated frequencies and we
use a fit transform method to model to the bag of words and create the feature
vectors we can then store the feature vectors in an array step for is to
create the classifier a classifier is a machine learning model that will be used
to classify whether a piece of text is positive or negative in this example our
classifier is a random forest consisting of one hundred trees are
random forest is a set of decision trees decision trees are graphs that model the
possibilities of certain outcomes so let's say a piece of text has a word hate
appear more than 20 times the probability that it's negative could be
something like eighty percent then based on other word frequencies we increased
or decreased probability accordingly
until we get to the leaf of the tree which will be a positive or negative
rating this is different from a standard regression classifier for data points on
a certain side of the line of best fit we can easily classify it. a random forest
tree is more like a series of lines one for every tree that segments are
possibilities once we've mapped all the lines onto the graph and we plot a new
data point or review based on its coordinates and then classified based on
whether it's in a positive or negative space it's time to test our classifier
are testing data so let's format the test data by cleaning the reviews and
creating a bag of words once we have our feature vectors for test data we can
move on to the last step the last step is for our program to correctly classify
the reviews in the testing data set as positive or negative will use a random
forests to make a prediction will then take the result and write it to a new
CSV file that's it let's run our program and see what happens
ok it printed out the first review that means it's correctly reading our dataset
then it's gonna clean and parse the training set create the bag of words
train the classifier then predict the test labels awesome let's test
first three predictions where one is positive and zero is negative let's see
the first three are 1 0 and 1 so positive negative and positive let's skim these
truly a masterpiece positive it's so awful that once you know ok negative
awesome looks like its performing sentiment analysis like a charm
sentiment analysis is still an evolving field of machine learning there are many nuances
and misspellings and slangs involved in human language that we haven't
really taken into account but we can with more powerful algorithms. Check out
the links in the description below for more information
and please please subscribe for more technology videos there's so much I
wanna make thanks for watching