There's a very important thing that I was keeping an eye on
as I was drawing up my training set of all the persons of interest, and
that is basically the size of that training set.
The reason that the size of your training set is so
important is that it usually has a big effect on the accuracy that you
are able to achieve with your classifier.
Let me show you an example of what I mean.
I once had a problem that I was trying to solve in physics.
I was using a naive Bayes classifier to try to identify certain types of
particles, and I had 1,000 events of each class in my training set.
And my question was basically,
is this enough events to capture all the trends in the data?
So here's what I do to answer that question.
I took my 1,000 events of data and I put about 800 into a training set and
about 200 into a test set.
Then I took these 800 and I split them again into 4 times 200.
So then by recombining different numbers of these slices,
I could come up with a training set that had 200 events, 400, 600, and 800.
And then I would always be testing it on the same 200 event testing set.
And what I found was something that looked like this.
The maximum accuracy that you can ever get will be 100%.
And usually in practice, the maximum accuracy that's even
hypothetically possible will be some number less than 100%.
And if you don't have enough statistics in your training set, if your training
set isn't big enough, you'll get an accuracy that's even lower than that.
So for 200 events, maybe I would get an accuracy that was around 55%.
However, when I go up to 400 events, my accuracy jumps to let's say 70%.
So I got another 15 percentage points just by adding more training data.
I didn't do anything else.
Then I go up to 600 events.
Again, my accuracy improves, but probably not as much as before.
So let's say this is 80% now.
And then when I add in my last slice of 200 events, I might be at about 82%.
And what this told me was that there was a trend in the data,
maybe something like this, that when I looked at it,
describe to me whether I could expect the accuracy could,
to get very much better as I increase the size of my training set.
Obviously, 200 wasn't ideal.
If I only had a training set that had 200 events in it,
I couldn't expect to get a very good accuracy.
By the time I got up to 800, I'm all the way up to 82%.
And moreover, by looking at this trend, I can see that it's starting to plateau,
and even if I went out to, let's say, 1,000 events, I might only get up to 83%.
So there wasn't as much of an advantage to add in another 200 events at
the end than this first 200 events at the beginning.
And so, one of my concerns, when trying to identify persons of interest,
is whether we'd be all the way down here, that we would have so
few examples of persons of interest,
especially relative to how many completely innocent people are in our data set.
But it's very hard to distinguish the patterns that set apart
the persons of interest.
Now, in a perfect world, I could maybe try to make a graph like this.
And if I found out I was down in this corner, I could go out and
try to find more data.
In this particular example, that's not really possible, right?
We only have as many persons of interest as we have.
It's not like more of them are just going to appear because I want to make
a better classifier.
But this is something that's very foremost in my head when I first tackle this
problem, is how many examples can we get of persons of interest?
So when I look at this list that I've compiled of the names of the persons of
interest and I see that we have about 30 people,
do I think that's enough to proceed with the project?
And the honest answer is, I don't really know, especially when I first started.
And there's no good way to know except to try it out.
But ideally, if you're working in a situation, say,
the self-driving car, where you have the option of asking a question like this,
how does the accuracy change with the number of training events, and
especially if you have the power to go out and collect more data if you need it,
this can be an extremely useful series of questions to ask.
And in general, more data is going to
give you a better result than a super fine-tuned algorithm.
This is something that surprised a lot of machine learning researchers when they
first started to discover it many years ago,
but it's become a truism of machine learning right now, that being able to
use more data will almost always help out the performance of your algorithm.