hello and welcome to this video on clustering words or tweets with R in
this video we are going to take a look at how to read text file and this text
file is going to be containing tweets and then we build corpus corpus means
collection of documents so in fact each tweet will be treated as a single
document and then we create term document matrix from that corpus so term
document matrix means we identify each and every term in those tweets or
documents and then we count how many times each word appears in each tweet so
once we create this term document matrix we also create a plot of most frequent
terms followed by hierarchical word or tweet clustering using dendogram so
earlier we had the use dendrogram for numerical data now we are going to do
this for tweets followed by non hierarchical k-means clustering of words
or tweets so let's get started by reading the file
for this analysis I am going to need a library called TM TM stands for text
mining so if you have not installed this package then we need to go to install
and install TM once this is done we run this library
and I'm going to set my working directory to my desktop where my file is
and then we use this command read lines so this can be used when our file is a
text file tweets.txt so that's why I'm using a read lines if this is a CSV
file then we say read .csv so let's store this in tweets so you'll notice
that there are 2,141 tweets if you want to look at them you can simply run
these tweets and each and every tweet will be listed here so it may omit some
of the tweets because of reaching a maximum print limit but all 2141
tweets will be there
now let us build corpus and we are going to store this in Corpus itself the
function is corpus and we'll make use of vector source and tweets so
now this corpus is ready with 2141 tweets
now let's create term document matrix let's store this in tedium tedium fourth
term document matrix but you can give any name you like and the function we
use is called term document matrix so we feed the corpus that we have created
just now and we'll also control will use this control
so we can say minimum word length equals something between 1 and infinity that
means it will contain everything but we can also restrict this minimum word
length using this control feature so let's run this
so now term document matrix is created if you simply select tdm and run this
it will give you some information that there are six thousand nine hundred
thirty four terms in two thousand one hundred forty one documents or tweets
and it's a very sparse matrix like lot of these cells are zeroes so sparsity is
very high so what we can do is if you want to reduce the sparsity
we can say remove sparse terms so in this tdm we can specify sparse equals
let's say 0.98 and we can store this new term document matrix in T so if you look
at T now you can see number of terms have come down and sparsity is about 93%
so there's some improvement depending on the data that we have we
can always play with this number to cut down on sparsity so once we are okay
with what we have so let's convert this T into a matrix and store in M so that
we can do further analysis
so now let's plot frequent terms let me store this in frequency freq and I'm
going to do row sums of M because it has many words like more than
100 so if we plot all of them it is going to be a very busy graph to cut
down on that what we can do is we can say that our freq is in fact a
subset of where freq is more than or equal to 50 so any word that has
appeared more than 50 times only that will be included so now let's make a bar
plot so I'm going to use some features so I'm
going to keep las at two and color will borrow color from rainbow
so we get this colorful bar plot where different words are listed on the x-axis
and the frequency is on the y-axis so obviously it can also be cleaned up so
we are not going into details of that so once you have identified what keywords
you want to include we can go to the next step which is create a dendrogram
to create a dendrogram we make use of dist function to first find distance and
we'll use scale to normalize and M so our matrix is M and let's store this
distance in distance
if you want to see this we can say print distance and with digits let's use two
so what it will do is it will print different words and also like calculate
the distance between the words in each document if the distance is high that
means those two words should not be in the same cluster if distance is quite
low then that suggests that the words should be in the same cluster once we
have calculated distance we can do hierarchical clustering hclust
using this distance and the method we are going to use is called ward.D
and let's store this in hc for hierarchical clustering so
we can run this when this is done we can plot this
so this is the cluster dendrogram quite busy because there are many words so
this is more clear so now you see that this is the distance and some words are
hanging up in the air so what we can do is we can align everything by doing
comma hang = -1 so this looks much better so we see some
clusters here like 1 2 3 4 5 6 7 8 9 10 so in fact more than ten clusters so
what I'm going to do is I'm going to put a red rectangle around about ten
clusters and that we can do by using a command rect.hclust
so our clusters are in hc and let's say we want to identify or group the outcome
into ten clusters so now if you look at dendrogram again so this is a huge
cluster so this is one two three four this has a single element five six seven
eight nine ten let's increase this to twelve and run these two lines again so
now that huge cluster is broken into one more and you see now twelve clusters
now to do non-hierarchical k-means clustering we need to first create a
transpose of the M matrix that we are using we will call this as m1 so you
can do transpose of a matrix by just using t so transpose means rows become
columns and column become rows let's fix a seed so that what I get here you also
get exactly same pattern for K means I'm going to assign a number 12 like we want
12 clustres so we'll store the results of k-means clustering in kc
so we use m1 and k if you want to look at the outcome I'm going to select kc
and run so it will give the output it shows that the 12 clusters have sizes
of like 652 116 and so on so it gives you size of each cluster and also
cluster means so for all the 12 clusters and for each word that we are analyzing
we get the averages so high average means that word has appeared in that
particular cluster with higher frequency so for example this one is 1.05 so this
is the highest for this column so this has appeared with higher frequency in
cluster 8 and then you have clustering vector so first one has gone to cluster
1 second one to cluster one third one has gone to cluster 10 and so on
so each of the 1141 entries also have information about what cluster they have
gone to and then for all the 12 clusters we have within cluster sum of squares
and then between sum of squares divided by the total sum of squares is about
33.8% so we want these numbers to be lower and we want this number to be
higher so these numbers will be lower if within cluster variability is low that
means the elements in each clusters are closer to each other and this number
will be high when cluster to cluster differences or distances are higher so
clusters are separated from each other more and more will lead to higher value
so this is 33.8 let's see if we go to only 3 cluster what happens
so you can see clusters to cluster distances or variability changes we want
actually clusters to be separated with bigger distances but that's not
happening if we go to only three clusters and in fact the within cluster
also has lot of variability you can see these numbers are quite high so you have
values up to almost six thousand if you look at the previous one with twelve
clusters so the highest value I see is about two thousand five hundred
ninety-nine so much below six thousand and most of the other values are close
to 500 or 700 and so on so with three clusters within cluster variability has
in fact increased so that's not good and also between cluster distances don't
seem to have improved with three clusters so that's why the percentage
has come down so this is how we can vary K and see the outcome and choose a value
of K which is going to work for us