Hello again!
And now, as they say, for something completely different.
The second half of this class is about document classification, this lesson and the next two.
And the only thing it has to do with the first half of the class is that both use the Filtered
Classifier.
Let's look at some documents.
Here are 6 documents.
They are very short documents (we'll look at a much larger example in a minute), just
a single sentence each, and they're classified into "yes" and "no" classes.
You can see when you read these that they are all about oil.
The "yes" documents are about oil coming from the ground, and the "no" documents are about
oil as used in cooking, "the food was very oily," for example.
We code this training set into ARFF in the standard way, with string attributes.
For string attributes we just take the text and surround it by quotes, just as I've shown
in the bottom here.
I've loaded this dataset into Weka.
We can just have a look at it here.
There it is, just what you saw on the slide.
And of course we can't do anything with this at the moment.
There are 6 distinct values for the text attribute, and no learning system can learn anything
from these 6 different values.
What we're going to do is use a filter, the StringToWordVector filter--Unsupervised, Attribute,
StringToWordVector--which is here.
It's got a bunch of options, but let's just apply it.
Wow!
Look at all these attributes.
We've got 34 attributes.
They're words like "crude" and "demand" and "the".
When you look at it, these are just the words that appeared in the training documents.
Actually, the type, the "yes" or "no" thing, has been moved to the first attribute, not
the last attribute.
When we look at the individual word attributes, like the one for "crude", it's just a number,
it's a numeric attribute with two values, 0 or 1: 0 if it doesn't appear in that document,
and 1 if it does appear in that document.
Let's go and classify this.
Let's use J48.
It's in grey, actually.
I can still select it, but I can't start it.
The reason why I can't start it is that, by default, Weka is predicting the last attribute,
and the last attribute is numeric, the word "was".
So I'll just change this to predict the type.
Then I can run J48, but there's a problem evaluating it, because there are only 6 instances
and we're trying to do 10-fold cross-validation, which isn't going to work.
Let's just evaluate this on the training set for the moment.
The most useful thing to look at in the result here is the decision tree that's produced,
which is here.
Let's look at the tree.
You can see that it tests on the single word "crude".
If "crude" does not appear, then it's a "no" document--that is, it's about food.
If "crude" does appear, it's a "yes" document--that is, it's about oil coming out of the ground.
It makes kind of sense; it's a kind of trivial example, I guess.
I'll just go back to the slides.
This is what we've done.
We loaded the data set into Weka.
We looked at the string attributes.
We applied this filter, which created a lot of new attributes, one for each word.
They were binary (two-valued) numeric attributes.
We used J48, had to set the class attribute, and evaluated on the training set.
Then we looked at the tree.
I want to evaluate this on a supplied test set.
I want to see what the predictions are on this test set.
These are the documents in the test set.
I've coded them as Unknown, that is, a question mark in the ARFF file.
We've never done this before.
We haven't ever looked at predictions for individual test documents or test instances.
Let me now go and get the supplied test set, which I have here.
Now I've got that test set.
I can start this running.
Well, it's obvious really--there's a problem evaluating the classifier, because, you know,
when I look at the test documents, it's an ARFF file with string attributes, and the
training documents are an ARFF file with word attributes.
Of course, I can take these test documents and convert them using the StringToWordVector
filter, but that still wouldn't solve the problem, because I might have different words
in a different order here, so I'd still have a different structure to the ARFF file.
We've got to do something different.
That's where the FilteredClassifier comes in.
Just going back to the slide, there's a problem evaluating the classifier.
We can't simply apply StringToWordVector to the test file.
The solution is the FilteredClassifier.
As we saw previously, the FilteredClassifier will create a filter from the training set
and use it for the test set.
That's exactly what we're going to do here.
Coming back to Weka, I'm going to undo the effect of this filter, so I've got the original
string attribute.
I'm going to find the FilteredClassifer (meta>FilteredClassifier).
I'm going to configure that to use J48 as the classifier, which is done by default,
and I'm going to use the StringToWordVector filter.
It's an Unsupervised Attribute filter.
Let me just run this.
Here we get the result.
That's actually not very interesting, because these documents had question marks instead
of classifications.
What I wanted to do was output the predictions, and I can do that in the More options menu.
If I click Output predictions and run it again, now I can see the predictions for the test
instances.
As you can see, there's 1 "yes" and 3 "no" predictions.
The actual class is a question mark in each case.
Coming back to the slide.
That's not exactly what I wanted.
The first instance is certainly "yes", oil coming out of the ground, but so is the third.
That should have been a "yes", and, in fact J48 has predicted a "no" for that document
"Iraq has significant oil reserves".
Obviously, it doesn't contain the word "crude", which is the test that J48 is doing.
Well, these are tiny little documents.
Let's look at something a bit more substantial.
I'm going to take a big dataset, ReutersCorn-train.arff.
Let's just look at it in a minute.
I'm going to open it now.
There are 1,554 documents.
This is a lot bigger.
If I apply the StringToWordVector filter, then--it just takes a second--I get a lot
of attributes corresponding to words.
Actually, there are 2,234 attributes.
Again, the class attribute has been moved to the top, attribute number 1.
I'm going to undo the effect of this [filter], because we're going to classify this using
the FilteredClassifier.
I'm going to set a different test set.
I'm going to open ReutersCorn-test.arff.
Then I'm going to run this with J48.
The FilteredClassifier.
It's just going to take a second.
It's finished now.
I get 97% accuracy.
Before we go on, let's actually have a look at what this dataset looks like.
I'm going to open up the file, the training file.
Here it is.
There are two attributes: a string attribute and a class attribute which is 0 or 1.
Here's the beginning of the first string, and it's a long string.
In fact, this open quote quotes right down to the closing quote here.
This whole bit of text is one string attribute value.
It's followed by a 0, which means the classification of that document is 0.
For this dataset, that means it's not about corn.
You can see this is regular text except these "\ n"s, those are new lines.
If we just had a regular newline in a string, then Weka would get confused when you tried
to load in that ARFF file.
It would think that the continuation of the line was the next instance.
So we just encode newlines as "\ n".
This is one instance, classified as 0.
The next thing starts with a quote.
This is the string, and it ends here.
That's a 1 document; this document *is* about corn.
That doesn't necessarily mean it just contains the word "corn", it means that a human has
decided whether this document is about corn or not about corn.
I don't know a lot about corn, but an expert will have made that decision.
These are the documents, and, like I said, there are 1,554 of them.
Each instance contains this extensive string.
If I now go back and have a look, well, I've got really high accuracy, 97%, which sounds
really good.
Unfortunately, though, when I look at this, the documents that are about corn, the "1"
documents--there's only 24 of them--and the accuracy there is 15 correct out of 24, which
is not so good.
For the "0" documents, the ones which aren't about corn, then I've got 573 correct out
of 580, which is very good.
When I combine those two, that's what gives me this rather high-looking 97% accuracy.
When I look at the tree--here it is--it's little bit more complicated.
We've got a branch on the word "corn".
If the document contains the word "corn", then we're going to look for the word "planted".
If it contains the word "planted", then it's a "0".
If it doesn't contain the word "planted", then it's a "1", that is, it's about corn.
Down here, we're looking for the word "1986/87", which is a very strange thing to be looking for.
We're looking for the word "maize".
Here we're looking for the word "the".
This tree doesn't look like is makes a huge amount of sense.
And yet it does get 97% accuracy.
This is what we've done here.
We looked at this dataset.
We applied the StringToWordVector filter.
We just had a look, and we found that there were 2,234 attributes.
Then we used the filtered classifier to get 97% classification accuracy, but we discovered
that the accuracy on the 24 corn-related documents was only 62%.
That's a shame, because those are probably the documents we're most interested in.
These are the ones that aren't about corn, and we get very high accuracy on those.
Which makes you wonder whether the overall classification accuracy is really the right
thing to optimize.
This is what we've done in this lesson.
We looked at string attributes.
We looked at the StringToWordVector filter, which creates one attribute for each different
word.
We looked at the options for the StringToWordVector--no we didn't look at the options.
Let's have a really quick look back in Weka here at the options for the StringToWordVector
filter.
Suffice it to say, there are a lot of options: it's a pretty comprehensive kind of filter.
We'll look at those options in a subsequent lesson.
We looked at J48 models for text data.
J48 is not necessarily a very sensible learning scheme to use on text data.
Then we looked at the overall classification accuracy.
Is it really what we care about? Perhaps not.
That's what we're going to look at in the next lesson.
Before you do that, you should do the activity, which will get you to actually do a little
more work on actually classifying real documents.
Good luck with that, and we'll see you in the next lesson.
Bye for now!