- Hi, this is Jeff Heaton.
In this video, we're going to talk about what it takes
to get into the top 10% of a Kaggle competition.
This is just the first in a series of videos about Kaggle
and how my research fits into that.
So be sure to subscribe so that you see
the entire series of how you can improve
your standings within Kaggle.
I'm not going to spend a tremendous amount of time
talking about what Kaggle is.
There's plenty of videos for that on YouTube.
In a nutshell,
Kaggle is a bunch of data science competitions
where you are competing with other data scientists
and machine learning people for prize money.
You can see the prizes at the right.
It's very very difficult to win one of those top prizes.
What a lotta data scientists do in Kaggle
is they're competing for standing within Kaggle,
Kaggle Master, Kaggle Grandmaster, Kaggle Expert,
all these various levels that you can attain
by participating in these competitions.
So let me lay my cards on the table.
I in Kaggle am what you call a Competitions Expert.
The next level up for me would be Kaggle Master.
All of Kaggle, you can see on this, I'm ranked 4,023
out of 94,000.
So what I've gotten is two of those bronze medals
that you see there underneath my Competitions Expert rank.
So I placed 225th in one.
That was my best one, Quora Question Pairs, 7%, top 7%.
You get one of those bronze medals for scoring
in the top 10%, and I've done that twice.
That makes me a Competitions Expert.
The next step for me would be Kaggle Master,
so I would need two silvers and one gold.
This video is more about how to get a top 10% win
because in all honesty, that's all I've ever done,
the top 7% in that one.
So I'll tell you how to actually win the thing
when and/or if I actually do that.
This video is simply an overview of my strategy,
how I roll with a Kaggle.
This is going to be a series of videos in a playlist,
so subscribe if you want to see the subsequent ones
and see how I actually do this.
This will be a more advanced Kaggle overview.
If you wanna see, there's plenty of videos on YouTube
talking about how to do the Titanic
and how to do some of the tutorials.
I'm going to assume you've already gone through
that kinda thing.
If you need a general overview of Kaggle,
just search for Kaggle introduction.
There's plenty of those videos out there.
You don't need another one from me.
So on my YouTube channel, I did my first ever poll.
And what 44% of you asked for, subscribers of my channel,
is advanced machine learning topics
related to my own research.
Well my own research, the stuff that is truly uniquely me,
Kaggle is my biggest hobby, I guess,
outside of my day-to-day job.
So my research, some of which I publish in journals,
is related to things that will help me do better
and better in Kaggle.
So we're gonna talk about that.
So since this poll gives you a chance to tell me
what you're interested in, if this video is interesting,
if you like Kaggle, you wanna hear more
about advanced techniques that I used,
at least to get into the top 10% and hopefully,
hopefully higher, give me a like, give me a subscribe.
That definitely sends a message.
So let's get into it.
In this video, I am going to give you
just a high-level overview of my strategy.
Then in the follow-up videos, we're going to go
into each element of that strategy in more detail,
and I'll tell you both what I did for these competitions
that worked and where I'm moving for the future.
So where I'm moving for the future is related
to areas that I'm researching, which is just
better techniques of feature engineering and evaluation
and ways of eliminating bias from some of these models.
But we'll get more into that later.
For now, we're going to just talk
about my high-level overview of how I approach
a Kaggle competition.
So if you go into one of these,
this is the Quora Question Pairs.
This is a pretty interesting Kaggle competition I did.
They basically give you two questions from Quora,
and you have to tell them if you believe
that the questions are asking the same thing.
Now in all Kaggles, there's a leader board,
and this leader board shows you,
so this shows you all of the entities competing.
Some of the entities are individuals like number seven.
A lot of the others are teams.
Color at the side is actually quite important.
Green means those are in the money.
So those people actually won money off it.
The gold colored ones are ones, including the green,
who would've gotten a gold medal from this.
So you can see the gold medals stop right at 16.
The number of people there that get gold,
that depends very much on how many people
are in the whole thing.
And then silver goes on for the top 5%,
and bronze goes on the top 10%.
It's not easy to get a top 10% and get a bronze.
That required considerable work, but neither of these cases
did I have to throw extreme money or computer power at it.
I did both of those just with my laptop.
So let's focus on that.
When you first start with a Kaggle competition,
you will do your initial submission,
and they will give you some sort of a score here.
This might be a Log Loss, this might be your RMSE.
Usually lower is better, but not always.
You can see looking at these top people
how very close they are together.
This is not always the case, but usually the top one or two
are very very very close.
So when you do your first submission,
and you can see how many entries they have.
So this individual entered 263 times.
There are a few Kaggle competitions where the winner
was just one. (laughing)
This is where somebody at the very very end
scored like the hole in one.
They used very very good cross-validation,
and they were able to evaluate their score
and they just submitted at the end and won.
Doesn't happen very often, but it can.
So as you go through the Kaggle,
and the Kaggles go between usually two and three months,
you'll see a lot of movement.
So if I am in the top 10% and I go to bed,
wake up the next morning, I might still be in the top 10%.
That's pretty common.
I might be in the top 11%, get pushed down,
or something cataclysmic might happen
and I might all of a sudden be in the top 25%.
So let's talk about what causes that
because that's a huge part of my strategy
and from what I've seen most other really good,
better than me, competitors who go very well in this.
I also haven't seen this covered a great deal
in other videos and tutorials, so let's talk about that.
So there are several key elements to your Kaggle submission.
There's the models that you build.
That's what most people start to think about.
You've got to build a model, a neural network,
a support vector machine, something like that.
There's the features.
That is how you transform the data
as they go into the models.
And that is very very very important.
Those first two though, models and features,
what you have to realize as you're competing in Kaggle
is that other individuals in Kaggle will be releasing
features and models throughout the whole entire competition.
If you wake up one morning and you have dropped
from the top 9% to the top 25%,
that means almost certainly somebody has released
a feature or a model that has knocked everybody down.
Because you were maybe here, they released some feature
or some model that's not incorporated into yours
or a lotta people's.
All those other people start jumping ahead,
and now you're pushed down.
The key to my strategy for Kaggle is being able
to very very quickly incorporate those features and models
that are being released continuously
throughout this competition into my own.
Because what I would see all the time
is I would get knocked out of the top 10%
that I wanted to be in, and as soon as I incorporated
whatever that feature or model that somebody has posted
onto the Kaggle forum into my own,
I just, I jumped right back ahead of all the other people.
Because I still had my own unique strategy that I had,
plus I augmented it with what had been released.
That is absolutely one of the keys to Kaggle.
You have got to be nimble, and you've got to be able
to incorporate those.
Now in subsequent videos, so subscribe,
you will be given some of the code that I use to do this.
And my code is designed very much around being nimble
and building these submissions very quickly.
So let's start to document my high-level strategy.
There are models and there are are features.
The models are taking the features and turning them
into your actual submission.
There's also ensembles.
Ensembles take multiple models and roll those together
to create another submission.
Ensembles are huge in Kaggle.
I will probably create a video, three subsequent videos
as part of this, that cover all of.
Now models, when somebody releases a new model,
what you need to do is very very quickly
get that model downloaded and reproduce the research.
Get the same result that they did
so that if whatever score that that model was purported
to get on the leader board, make sure you can reproduce it
and get it.
Then you take its submission file.
The submission file is its correct answers
to the Kaggle test data set that they've given you.
And you will take that, then you will ensemble that
with your existing model.
And I'm going to give you the code for my ensembler.
So that is one of the first things that I do.
If they have a model, I have a model,
we ensemble the two together, and usually,
unless their model is virtually the same strategy as mine,
which is not usually going to happen,
the ensembled score of those two will be higher.
Then features, you'll see features released
all over the place.
You've got to scour the Kaggle forums as this is going on.
Usually you'll know when a new feature
or new model comes out, your score will drop
as everybody incorporates it.
So now you've got to incorporate that feature.
So that's essentially a new column that you're going to add
to your own model.
Now if you've got models from other people,
you might want to augment them with that new feature
that you know about.
And then you're also creating your own features.
The features then go into your model.
That model then goes into the ensembler.
So I have code in my Kaggle toolbox that does the featuring
and the ensembling.
So you'll see that in subsequent videos.
Then I also have a script that I run.
I call it my combiner.
But basically what it's doing is it's taking
all of those features, so I have features from
that I found from other competitors that they've posted,
I have features that I created,
and I need to take those features and combine them all
into one file.
So I like to keep all my features as CSV files,
and this, the combiner just takes a bunch of CSV files
and stitches 'em all together so that you get
the combined columns of each of those.
And then I also have switches in there so that I can trough
certain of those columns.
You don't want to be generating your features
every time from scratch just to build a new training set,
or you're going to be spending all kinds of time doing this.
This is one of the key ways that I do this
just on a laptop.
And then I have an evaluator, which is what I use
to look at those features that the combiner
is putting together and get an idea which of those features
are important, which are not.
So this is very much a flow chart,
and in fact, in the next video, I will draw this out
as a sort of flow chart so that you can see
how this is all fitting together.
And then we'll start looking at each of these pieces,
and I'll give you the Python code that I use
at least as the starting point for when I do
a new Kaggle comp.
So there you have it.
This is the high-level overview,
and there will be more videos coming
to show you how to do each of these individual steps.
So if this was useful to you, subscribe and like.
Thank you.