hello everyone Ken here back with a new data science video today I wanted to do
something special I saw that there was a lot of really negative reviews of
Captain Marvel on YouTube and some other resources and I wanted to see what the
crowd thought what the Twitterverse thought so I am doing a sentiment
analysis on Captain Marvel I'm also comparing it to Avengers
infinity war in context if this interests you if you liked the video
please hit that like button and if you'd like to see more like it please
subscribe I do this analysis in Python and a couple of the modules are very
helpful and actually getting this doc so the first module that I'd like to give a
shout out to is Twitter scraper so it's more simple than Twitter trust API you
don't need any login or credentials or anything like that and it also gets
through some of the limitations of that API where you can go back in time as far
as you want which is really cool I also use the Bader sentiment tool which it
takes a lexicon approach to sentiment analysis so it has a big dictionary of
words and it attaches sentiments to each one of those and also based on the
context that the word is used it gives it a sentiment score there are
a couple different levels of score so they mark how many negative features
used how many neutral features you use and how many positive features you use
and that's all combined into one composite score bitter is one of the
best practices in the industry it's not perfect some things do slip through the
cracks but at a large scale it works very well and I'm excited to actually
build this into my analysis I haven't had a ton of exposure to sentiment
that's why one of the reasons why I'm actually doing this because it lets me
fool around with some new tools and to explore a very interesting topic you can
find three interesting links in the description below the first is a medium
article that highlights well that was actually the inspiration for this video
that talks about using sentiment analysis to analyze the anthem game lock
the second is another medium article about the details of how they
that works and the last is a link to my github where you can find the entire
code base for this analysis so let's just jump into how I set up this
analysis I'm not gonna go through all of the code but again if you want to see
that that is in my personal github the first thing we do is import all of the
relevant modules pandas were going to be using because we're converting a lot of
this into a data frame to analyze it date time we're gonna be using to filter
by the specific time of day and I talked about Twitter scraper and Vader
sentiment analysis we're also using lang detects because we only want to evaluate
english-language tweets Vader does not work as well in other languages from my
experience and so that is something that we really need to do to get the most
useful analysis possible we're also including some visualization tools mouth
portlet and Seabourn to make it look pretty I created this detector function
because for the language detection we need to make sure that it is a texturing
that it's this big fit in if you feed in a number or something that is non text
you will get an exception thrown so this is just saying that if there is an
exception we put none in instead then we build this analyzer object and create
some of the parameters for us to query the tweets I'm looking at you know three
or four days before the premiere and until now basically after the premiere
so we will right here actually end up queering the tweets this takes 10 or so
minutes it takes a while we then put them into dictionaries so that we can
put them into a data frame finally down here is where we actually sort for the
English language now I would recommend saving these to CSV files your data
frames now because it'll save you some rework in the future if you you know
mess up Python crashes you will have that saved so you don't have to run the
Twitter query again which is fairly time-consuming
now let's jump into the actual sentiment analysis so right here this is pretty
much all we have to do we apply our analyzer clarity scores to each of the
text segments in the tweet and we get those four features that I've mentioned
before the amount of negativity the amount of neutral nests the amount of
positivity and the composite of those things which is your actual sentiment
score we then append that to our data source and I looked at a lot of these
tweets and it looks like there were a ton of duplicates so those were
advertisements or things along those lines and I wanted to get rid of those
so we get a true or feeling of what the people really thought so I dropped the
duplicates here finally I was having some trouble filtering by date time so I
switched our timestamp into day time and made sure we were only getting after
premiere tweets that were actually after the premiere of midnight the day after
the movie came out after looking at a histogram
I saw the majority of the tweets had a sentiment score of zero which means that
they weren't necessarily negative or positive for our analysis sake I removed
all of those tweets because I wanted to understand the actual parity between
before the movie and after the premiere had launched needless to say I was
pretty surprised by the results the even before leading up to the movie it had a
fairly positive sentiment score of 0.3 for and after the movie premiere the
sentiment score actually went up to 0.42 that's almost a 20 percent increase
which is fairly impressive it suggests to me that a lot of the Twitter
community at least enjoyed the movie even in spite of what has been said on
YouTube and some other mediums are in this same analysis on Avengers infinity
war from last summer and I also got very interesting results Avengers before the
premiere had a sentiment that was positive but lower than Captain Marvel
of right around point one for after the movie had
premiered the sentiment dropped to point one that doesn't surprise me a ton
because of how controversial the ending of that movie was but it still surprises
me that in general all of the sentiment was lower
now I'm not qualified to make any claims about the quality of Captain Marvel
but this analysis suggests to me that the general Twitter community didn't
feel too badly about it that it was fairly well received in spite of what a
lot of people were saying about it and the ratings that it got thank you so
much for watching and please let me know what you thought of the movie in the
comment section below if you have any questions about my
analysis please leave them there as well until next time