Welcome everybody you're watching mr. fugu data science.
Today, we will be doing basic natural language processing using R studio, for
our online sales data project we're continuing. As always the code will be
linked in the description below. We will use Kwan Teta (quanteda). Today, for our NLP
processing: the outcome will be separate some sentences into individual words,
remove punctuation, stop words, and numbers. Generate word frequencies, do
some plotting with word cloud, and frequency graph. As of today May 29, 2020.
Inside R studio, using anything less than 3.5. You will have issues installing
quan tee da (quanteda). I had 3.4 dot something. I had to update
R studio. I had to update R, and then I had to
update all of my packages. Then since we're using today
Jupiter notebook. I had to reinstall the eye r kernel (IRKernel). Inside of R studio. So
we can interface with Jupiter notebook. If you need to update your anaconda, I
suggest you do that now. and choose the option of the developer tools install
inside of R studio again. Now, inside of jupiter notebook: i had to reinstall
packages that i had before. Because, i updated everything i don't know why i
didn't connect but that's the case. So now I have version 4, instead of 3.4.
We need to do our installs of quan tee da (quanteda). Knitter and tidy verse.
and you do your libraries. I've never tidy verse of quan tee da (quanteda). As always and
then we'll get into the data. Since this is something we're progressing. I'm going
to skip this, and actually let's do this. Let's refresher on what the data look
like. Continuing from last time, what we did was remove any orders that had
cancellations. As well as the counter part which was the original order. This
is the rest of the data that was generated or we got rid of that. Okay.
Well that's actually down here but, that's what the data will look like.
Since, we're taking one specific column for the item descriptions. This is
what some of them look like. You'll have a string of words for each item
description. What we need to do is convert it to lowercase; where I use an L
apply function. I took our column and called the specific method or
function of setting it to lowercase. I stored that as product description
lower. What I needed to do from there was split our sentences into words, okay.
So then I do to string, and then what we just convert it to lowercase. All right,
then we need to create something called a token; okay,
where we're taking what we just split and we're removing all of the
punctuation, the symbols, as well as numbers. And that's what that looks like
here. Okay, from this we generated quite a bit of words. But, now we have to start
moving on to frequencies. But, in order to do that; we need to remove stop words. And
a stop word; is something like and, the, with, etc. All right.
So we remove those, and we say: token remove our stop words, from the remove
punctuation we created previous. Now you notice a difference; you don't have
I, you, don't have you you don't have things like this. Now, we're getting a
little closer. But, if you investigate these data and do your plotting: you'll
notice something like you need to remove, and then you have to go backwards. So
then I say okay, now I need to do a remove of specific words and that's what
I called here. Because, I want to use something let's descriptive, it means
something to me. While I could keep these and I could say Oh people liked a lot of
pink things. That doesn't help me out as much. So I'm removing those here. Okay, so
then if you notice we don't have the word color the British word of color
spelling here. We then have to create what's called a DFM: document feature
matrix. Where we're taking our tokens, and converting this to a matrix of document
features. Okay, so what I do is I call DFM and I take what I just called our tokens
with remove stop words. And that's what we're generating here. If you notice
you start seeing a frequency count of each one of these words. So we're getting
closer, which is great. And if you notice we have two thousand and one unique
features. Each feature represents a specific word. okay
also you notice that I don't have a sparse matrix here. The reason for that
is I did not retain the space, that was reserved for the word I removed. You can
keep that, and it would just represent a zero in that space.
But, in this context I neglected that because it just wasn't necessary because,
I won't evaluate the words have removed for any analysis. Then we used a DFM
select where we're removing from our DFM based on some specific pattern. For
instance here, if you use the pattern of minimum N characters. That says that
you're taking a word that has at least five characters in that word. You don't
have to do this but, I'm just showing it because in some circumstances that may
be needed. Here we're not using it but it's extra tool that you can use. So I
need to say hmm this look at our top features and let's get that frequency.
And when I use top features and I say well let's look at the top 10 items by
frequency, and that's what's printed out here. You can also select by a threshold
value: where you say okay, let's get the minimum frequency based on anything
that's at least 13. Then you could also consider well, what if I wanted to
do some grouping by words? And that's something we could look at later you can
look at from the word you're evaluating would other words or around that word in
your sentence: like before or afterwards right. You could look at things like that
depending on like if you're analyzing let's say like tweets or something from
the news, you know that that would come in handy. So we set up a word cloud plot.
It's really simple here: I looked for something like add at least a frequency
of 13 or more. I put our DFM descriptions and then I call the text plot underscore
word cloud. and I set a random seed. If you notice here all these big ones
right here: occur the most. So it appears that in our online shopping data. People
are looking for vintage things, bags, cakes, something that was related to the
holidays, you know so hearts and Christmas apparently. People like retro
items. You know so this is really cool because, it jumps out it pops at you. It's
really descriptive you know. It's something like you plot and you can
instantly see what's going on. You can also do a bar plot, and make a little bit
of a let me get this a little smaller so we could see this. So here what I
wanted to look at was say we're going to use the text that frequency method lets
or a function that's within the quan tee da (quanteda).
This allows me to take 20 of the top items by frequency here. So these are the
top 20 items and the relative frequencies. You could input whatever you
want top 10 top on 100, whatever you want. nd then I'm using the de plier
method (deplyr). Where I can call ggplot and I say well let's order this from highest to
lowest. Where the y axis is the frequency. But, we're flipping the coordinate system.
So instead of these bars being vertical they become horizontal. so it's a little
easier for us to interpret. You just call the stat identity,
and then your labels you're taking everything for the X here. Then
we're calling the top-20 here for our Y with our frequencies. Then I change
the theme here so I can increase the size of our text. So could be legible
which is what we want. I ran through this very quickly because, this is
something that you could pause the video stop at home, work on this on your own,
get an idea of what's going on and plug in your data into these these plots. And
figure out. What's going on it's very useful for an interpretation for a
project, for a presentation. I would like to say thank you for watching this video.
Please subscribe and turn on that notification bell. Throw a like if you
thought this was useful content. If not leave me a comment and tell me why, so it
could be beneficial for all of us. I would like to make a better channel if
possible. Have a good one see in the next video. Bye