Hey Marla! Oh hey guys! Hey Graeme. Hey, Ingo, Ralf, how are you today? Ingo, it's time for...
Five Minutes with Ingo? Five Minutes with Ingo! Today also with Ralf, co-founder of
RapidMiner and a true expert on text analytics. Just talking about Sentiment Analysis for
Data Scientist #7. Ralf, how is that working? Well, we've seen text - we have classification
in the past, so now we want to apply that to text. The question is how do we do that.
Well, let's look at some statements. What do people say about Data Scientist #7? OK,
cool. I just happen to have, like a statement here. Oh, how convenient. Yeah, it is. So,
We have for example a positive statement: 'Unicorns are amazing.' Yes they are. Yeah
of course they are. A lot of people have some trouble, so let's check out what they are
saying: 'Finding Unicorns is Difficult.' Indeed it is. OK, so those are two text sentences,
two, well yeah, unstructured data, that really is. How can we bring this into a structured
format? So, Ralf, what would the first step look like? Well, grammar is always so difficult,
and I think the most important part is in the words, so let's skip grammar. That's,
basically pretty much what I'm doing in English every day. True, yeah, well. Anyway. OK, so
let's tear the text apart into new components. So, we are like, like just breaking this down
so we have still those two sentences, but every sentence is broken down into what we
call tokens. So every word is becoming a token. OK. So what's the next step? Well, there's
still a lot of words, and some of them don't carry as much information, like 'is' and 'are',
are not so central. So let's get them out. OK, so we throw them away, and that's what
we call 'stop words', we can remove them. Well, now we still have the two sentences.
It's not really as structures as I would like it. We're heading towards a table so let's
put in more structure. Since we put away the grammar, we ignore the word order, and basically
only look at the words. OK, so that is interesting. That almost looks like a good structure already.
We need a pen. Whiteboard #1, could you give us a pen? Alright, perfect. Thank you. What
a good whiteboard. So, we could actually say like those two words are in both sentences,
so we can actually make a cut here, and another one here, and another one here. So it's almost
like columns in a table. We have almost a structure now already. OK, so then what can
we do now? Well, you want to simplify things so we just count is a word occurring or not.
So here there's no word here. True, but we have one here for 'finding.' Yup, and one
for 'amazing.' And for 'unicorns' actually we have two ones. And for difficult we have
a simple one. So look at that, now we have actually a table. Every token is becoming
a column. The values here is just the count of words for every word in each sentence.
Now we can add another column, which is basically the positive sentiment or the negative sentiment
here. And now we have this label that you usually want to predict with machine learning.
So we can now use any machine learning method. SVMs by the way are great for that, by just
taking this table here, train the model on this data in order to predict if it's a positive
or a negative sentiment. And that's really about it for all of the general idea about
text transformation. The only problem is: what are you doing if the texts are becoming
any longer? Well, the longer the text the higher the values will be, so it's kind of
unfair. The longer texts are stronger, so we want to divide that by the lengths of the
texts. This has two words, so I divide those by two. The other one has three words so lengths
is 3, I divide it by three. So afterwards that does not depend on the lengths anymore.
So that means really, 'unicorn', for example, is more typical for the first text, because
it's occurring in 50% of all the words basically. And here is only in 33% of all the words.
So that makes it a more typical word. Unfortunately it occurs in both texts so it doesn't make
a typical word for a positive or negative. But for example 'amazing' and 'difficult'
you can see there's some difference. I can. OK, that is excellent, that is really amazing.
Now of course there's another problem, in my opinion, because what are you doing with
tech words which are just very frequent overall, like we've been throwing away those stop words
like 'is' or 'are.' So those are words which are very frequent in all text documents, so
what can you do about them? So this was, it's called text frequency. It's about how often
is the word in the text. Oh, I think we needed a second whiteboard. Where's whiteboard #2?
Here it is. There we go. So this is text frequency. It's the count of the word divided by the
length of the text. And then the other term is taking care of words that are two frequent
in two many documents. So we count in many documents they are, and take the inverse of
that. So that means we are normalizing the term frequency. So the term frequency is what
we see here, and by normalizing this, we give terms which are, well, very frequent in all
documents a smaller weight. And that's a perfect representation for text documents in general,
and that's really a great way to transform unstructured information into structured information.
And that's it for today! Interesting, thanks Ingo, thanks Ralf. No problem! And this has
been your Five Minutes with Ingo... and Ralf!