Hello again! In data mining, people are always asking "how much data do I need?" We're going
to show you how you can address that question in this lesson using learning curves.
The advice on evaluation from "Data Mining with Weka" was: if you've got a large, separate
test set, then just go ahead and use the test set.
If you've got lots of data, use the holdout method.
Otherwise, use 10-fold cross-validation --
it's the best way of getting the most reliable performance estimate out of a limited amount
of data.
You might repeat it 10 times or more, like the Experimenter does.
But how much data is a lot? Well, that's a good question, and there is no answer:
it depends.
Supposing you've got 1000 instances.
That sounds like quite a lot.
If you've got a 2-class dataset with 500 of each class, then maybe that's pretty good.
If you've got a 10-class dataset with 1000 instances and the classes are unevenly distributed--so,
maybe for some classes there are only 10 or 15 instances--well, that doesn't sound so
good.
Although perhaps you don't care about those small classes.
It depends on the number of attributes.
Again, with your 1000-instance dataset, that sounds like a lot, but if you have 1000 attributes,
that might not be such a lot of instances.
It depends on the structure of the domain.
Are you looking for complicated decision boundaries? It depends on the kind of model, the sort
of decision boundaries it makes.
If you've got a machine learning technique that looks for linear decision boundaries,
then they're pretty simple.
You might not need so much data as you would for ones that look for more convoluted linear boundaries,
or for decision trees, perhaps.
It's an impossible question to answer.
The only way to look at it really is to look at it empirically using learning curves.
I've shown a plot here of a learning curve.
As the size of the training data increases, the performance gets better and better, but
of course, it asymptotes off.
The point where it starts to asymptote off is probably enough training data to get a
reliable estimate.
Let's talk about how to plot a learning curve in Weka.
We're going to sample the data.
When you do sampling, we're going to choose a sample, and you need to understand the difference
between sampling with replacement and sampling without replacement.
When you sample, it's really a question of whether you move or copy the data.
If you sample "with replacement", then it's like you take it out of the original dataset
and put it into the sample dataset, and then replace it back in the original dataset.
You don't really take it out.
You copy it from the sample data, the instance from the original dataset to the sample dataset.
"Without replacement" means you move it.
You can't see it again;
you can't sample it twice.
If you sample with replacement, then instances might occur more than once in the sample dataset.
If you sample without replacement, then they can't.
That's the first thing.
We're going to sample the training set, but not the test set.
We want to find out how performance changes as the size of the training set increases.
But the test set determines the reliability of our estimate --
we don't want to make that artificially smaller.
We always want to use the same size test set.
We can do that in Weka by using the FilteredClassifier.
There's a Resample filter, and if we wrap that up in the FilteredClassifier, that means
that the filtering will apply to the training data and not to the test data.
I'm going to do that with the glass dataset.
I've opened the glass dataset here.
I'm going to go to Classify.
In meta, I'm going to find the FilteredClassifier.
Then I'm going to check--I'm going to use J48 as the classifier.
For the filter, I'm going to use the Resample filter.
It's an unsupervised instance filter:
we're resampling instances.
There it is.
Here are the parameters.
We can sample with or without replacement, and I would like to sample with no replacement,
so I want to make that true.
I want a 50% sample.
I can go ahead and run that.
I'm doing 10-fold cross-validation, sampling the training set,
using a 50% sample of the training set and leaving the test set untouched.
I get 65% performance.
Back to the slide.
Here is the 50% level:
65% performance.
I did this for other sample sizes,
which enabled me to plot this learning curve empirically:
the performance against the percentage of training data I'm using.
I've shown the ZerioR performance there, for reference.
The line's a bit jagged, and to get a smoother line, I'd want to do it several times with
cross-validation.
If I do 10 repetitions of J48, then I get this line here.
(I did this with the Experimenter.
It's very easy to do.) Then I did 1000 repetitions.
I get this red line here, this smooth line.
You can look at this line and make your own judgement as to how much training data you
need to get pretty close to the ultimate accuracy of J48 on this dataset.
It looks like providing you have about maybe 50-60% of the training data, you're going
to be fairly close to the final accuracy.
That's it for learning curves.
The question is how much data is enough? The answer is we don't know! So, you can plot
a learning curve.
We looked at resampling with and without replacement, but we didn't want to sample the test set,
because that would just decrease the reliability of evaluation.
We used the FilteredClassifier.
Obviously, the performance figures you get are only estimates, and you can improve the
reliability of those estimates by repeating the test several times.
That's it for now.
Off you go and do the activity, and we'll see you again in the next lesson.
Bye for now!