- Hi, folks.
I'm Imran and I'll be talking to you,
together with my teammates Alvin and Iman
about how to do AI assisted featured selections in Spark.
And be sure to give us feedback.
We would love to hear from you.
So I'm gonna walk you guys through the introduction.
Why should you do feature selection?
What are the current approaches
and the limitations and how is our approach different?
And then I'm gonna hand it over to Alvin
who's gonna go into more details about our approach.
And then lastly we'll talk about our open search library
that you can use to do this yourselves in Spark.
First, something quick about Clarify Health.
We process about 11 billion healthcare claims
for 150 million patients.
It's a little over 40 terabytes of data.
We train hundreds and hundreds of models.
Right now we're at 2,000 models
and they contain about 2,000 features.
Everything is built in AWS in Spark
and you can,
within five hours we can train models
for 84 million patients.
So the AI assisted feature selection is part
of our Auto-ML pipeline.
Or how I like to call it,
how can new models be created while I'm sleeping?
So we start out by acquiring the data certainly,
the second step is our AI assisted feature selection.
So this is where we'll talk about today.
We're able to automatically selected which features
should be included in the model.
After the features are selected then we go through
a selection of the algorithm to use for creating the model
and after that we do
an automated Bayesian Hyper-Parameter tuning
to find the appropriate parameters for that model.
And then next we do the training of the model.
And then after we have trained the model,
we have a set of automated tests that run
that are checking various things in the model
and figuring out if the model is good or bad.
If the model is bad then it gets put into the rejected queue
where a data scientist will go in, take a look
and figure out what went wrong.
If the automated test pass, then the model is promoted
to the model tracking and then after that into
the model explanation stage where we generate
the explanation for the model and we also generate
the metadata of it to be able to do exploration.
And then these models become available in our interactive UI
that customers can use to interact with the UI, see
the predictions and play with the models.
And our main goal here is no human involvement required
unless there's a model is rejected
and the data science needs to go look at it.
Okay, so let's talk about are more features better or not?
So certainly, there's a lot of people who are still
have models with less than 20 features.
If that's true, then this is probably not the talk for you.
But if you're people who have more than 20 hundreds
of features, then certainly less features is may be better.
So let's talk about some of the problems
with more features.
So first is that more features make
the models hard to understand.
So if you've ever looked at coefficients for 2000 features
or feature importance for 2000 features,
you'll understand that.
It also increases the training time,
the more features you have, the longer it will take Spark
or anything to train the model.
And then it introduces noise into the models.
So as you have more features,
the models is not able to find the optimal parameters.
And you also increase the chance
of having highly correlated features.
So this can confuse the models in turn,
resulting in a sub optimally tuned model.
And lastly, more features can result
in overfitting so your model is not able to generalize well.
So it's something I would say less is better
if you can figure out what the right set
of features are to use.
Okay, so what are the common current approaches?
Right, so the first approach people take
is what I would call the domain knowledge approach, right?
So someone will say, well, I know that diabetes people
don't stay longer in the hospital.
So as a result, we should not include diabetes as a feature.
What we found in the past is really a lot
of times our domain knowledge is not that good.
Once we look at the data,
we find that a lot of these domain knowledge
that we hold is not actually correct.
Second thing people try is they remove features one by one
from the model.
So you will remove a feature, see how the model performs
and see if that feature was really useful in that model.
That works if you have a few features,
but when you have hundreds of thousands of features,
the number of possible combinations is extremely large.
And there's just no way you can try that.
The last approach people tend to try
is just use all the features.
I mean, we spend time building these features.
Let's just throw them all in and let
the machine learning models figure this out.
This also results in suboptimal models
because the model training is not able
to figure out everything for example,
the difference between one hot encoding
and numerical encoding is really something
a human figures out and says do the numbers
means something, or is it just unique values?
This is also something I would call everything
but the kitchen sink approach.
So the ideal approach we've seen is
that it's combining the human and the machine together.
So this is what we call the AI assisted feature selection.
When we thought about this, we said, okay,
what is it that this would need to do?
So the first requirement was, it had to be automated,
and had to intelligently select features.
It wasn't a random selection of features.
Number two was it also had to scale.
It had to handle thousands of features like we have
and at least 50 million rows and 40 terabytes of data.
And the last one is, we also wanted the ability
to preserve certain features that we always want.
Customers might wanna see age,
even though it may not be the best feature for the model.
So these were the main requirements.
I'm now going to turn it over to Alvin
who's going to talk about how he
and the other team built the our approach
to be able to address these.
This is Alvin Hendrick and thank you Imran
for giving such a nice overview,
why we chose to use the structure learning approach based
on and then, we our goal was to basically come up
with an approach which Imran already described, right?
To basically get rid of thousands of features which we have
and focus on the features which we really need to focus on.
We chose an approach called a structure learning approach,
which is a filter based approach, right?
There are wrapper methods out there,
tailored for very high sparsity,
scalable and efficient sub-optimal feature selection, right,
An option to specify must include features.
So we were looking for solution
which basically fits our needs, because of the needs
which you see on the screen must_include_features.
We just don't want to get rid of all the features, right?
So the steps we have is like quick ranking
of all the feature importances, no model training needed.
Then we do a linear search based
on the rank feature and then train on N cross validation
step model to basically get the best model
out of those features.
So here's our project looks like.
So you see on the bottom of the screen the feature set,
this is the whole feature set one
to n, right hand side you have an outcome and label
and then we have the rank feature on the left hand side,
which is a yellow box.
So the way we calculate is basically we calculate
a similarity between the feature and the outcome label
which has the highest correlation.
And once we find that highest correlated feature,
we put it in the ranked bucket with a score
in front of it.
So you see that the feature key is now
in the ranked feature bucket, right?
Because that is highly correlated with the label outcome,
which we figured out the similarity
with the highest correlation.
So this is how we move towards the next feature.
So now we pick a feature_two.
If you see on the bottom of your screen, right?
that feature should have a highest correlation
to the outcome label,
but it should have the minimum correlation
with an existing feature which are already
in the ranking feature bucket right?
So, we are going to penalize using the formula
which you see on the screen, right?
It's the basically similarity of the feature I
with an outcome of label
and then taking the max of similarity of feature I
in conjunction with the feature already
in the ranked bucket right.
So, we choose to max or average so that the scoring
is independent from the number of features
that are already selected, right?
So, all we are doing is we are taking
the highest correlation with an outcome label
and making this feature to deeper ties
because there is another feature which already have
in high correlation with an existing feature
in the ranking bucket.
Right, so feature_2 gets there and it gets
the rank number two as you can see.
So, here's the algorithm.
I'll describing the quick steps right.
So we rank feature based on their coalition
with an output, select the feature
with the highest correlation score as top rank
and remove it from back of unranked feature.
Ranking the remaining feature based on this formula,
feature with close absolute correlation
of features outcome minus a max of the feature
with an already ranked feature in the bucket.
So selecting the feature with the highest feature score
as the next rank and removing it from
the bag of features, right?
We repeat this process until all the features
are ranked in unranked bucket.
If there is a must include feature in the list,
we put them on the top of the already ranked features list,
because we want those features to be in the model
Here's the benefit of our ranking algorithm.
So rank feature based on the correlation with outcome.
If a feature significantly correlates
with previously rank feature, it gets de-prioritized, right?
As I said earlier, even if it has high correlation
with an outcome, right, the ranking is supervised
but without the computational burden of training.
So here's an example.
So we have feature_3 which is highly correlated
with feature one, the selection of one
of them will be suppressed the selection of the other
as the core cross correlation is very hundred percent right?
So, the selection of one of the features will be suppressed
than the other one because they are highly correlated.
So, both will give us the same answer
or same benefit for a model.
Now take an example two write where feature_3
is correlated to feature one and two, right.
But even the feature two was correlated, right?
It will be like you can have a feature,
second one can get deprioritized, right?
Or the third one gets prioritized, right?
Because of the high correlation, right?
And then what happens is,
here's an example, we chose a Boston data set
where we had all the features, right?
So the first experiment to demonstrate we run it
with no constraint,
which means we don't care what features to include,
we let the algorithm figure it out.
So here's the ranking of the features,
like you get the LSTAT, PTRATIO, right rooms, CHS right?
These are the couple of features right?
But what we did in the next experiment,
we choose text and in this features to be included
in the list because we are going
to rank them very high because we want them
to be part of the model right?
And then you see the list on the bottom changes,
but it doesn't change as much because
if you see LSTAT, PTRATIO and RM was on the list
on the top list.
Similar thing goes with the second experimentation
where RM, LSTAT and CHAS are still on the top
of list plus the other two features gets included
as well, and the others follows, right.
And then what we did was it was just an experiment ran
on two sample to demonstrate the number of tress 20,
and MaxDepth five.
So here's the simple graph to demonstrate.
So if you see on the bottom left of your screen,
this is the must include feature we started with
the R square is just about point four, right
on the left hand side.
And these are the number of features we included.
So the default we started with number of features
included is the first two features.
And all of a sudden on the top you see
as soon as you get the four features into the model, right?
You see an elbow there and that's what helps us
to detect when we could stop doing more feature selection
or we have reached the peak where we can get
that's the best R squared
for this model training we will get.
We don't need to look into the other features as needed.
But if you see when we don't give any opinion,
it starts with the R square of point six.
But eventually it goes to point eight, right?
Or somewhere around that, right?
Not more than that.
So the algorithm is trying to figure out and it's trying
to do a feature selection and in an optimal way, right?
And luckily it has a capability
to include those features which we want.
So here's the approach and incremental training
before sample the training data to our desired size.
After extracting the feature ranks,
we start model training by one to N features in step
of s feature per iteration cross validation for each model.
At each step, the model accuracy is estimated.
If s equals one it means that all the features
are gradually added one by one.
A larger s means that the multiple line features
are added at each step right.
So we are trying to
once we have ran those features,
you're trying to train the model with the rank features,
and we have capability in the algorithm to choose
how many features you want to include
and train the model to get the best model out
of those ranked features.
So the end result was that plot which we already saw, right,
and using the elbow detection, we find the optimal number
of feature to be selected for this model.
As we saw the number of features for the Boston data set was
we need to just select four features at the max
to get the same accuracy.
So it is benefits of our offer is fast and scalable,
quick feature ranking, no model training needed.
We just need a correlation matrix,
a linear model training steps for each cross validation.
You can manually select specific feature to be included
in the modeling application requires them to
which is a very important thing for us
and select other features to complement them.
Scalable implementation with spark,
application on big sparse data sets successfully reduces
our features 30% while maintaining the accuracy,
so that we can reduce our feature set
by the 33% while maintaining the same accuracy
as we would have got with all those thousands of features
in the model.
Here's our open source library.
So we'll be open sourcing our library, thanks to Imran,
he just really worked very hard on the test,
to make this thing happen.
And so we decided to open this source this library,
and then it is applicable and available both
in Python and Scala.
So here's the two methods, you will see.
Notice, here's a simple API.
First is the feature ranker, you pass in your data frame.
You pass your feature columns,
and then you get the output column and you have
to have a must include feature list you want to include.
And then we have a feature selector.
From the model like figuring out from the rank feature,
what are the best feature we could use
to basically train the model finally.
Here you will see a feature inclusion increment,
which is a very important parameter.
So it was like it allows you to do a jump whether you want
to try each and every rank feature and then train the model
to basically extract the best R square of the model
or best fit.
Or you can do a set of model like I want to try
from these rank features,
I want to try three of them in incremental of three.
Or like the first three will be included, the next three,
the next three, and then it will figure out.
And then you can obviously specify
the train test split ratio, cross validation,
if you want to do an evaluation matrix
or we want to evaluate against r square
or an RMSE or ME.
Here's a simple test to test it using the Boston data set.
So you see the first thing is
to basically do the feature ranking.
Once you have got the feature rank,
you call the feature selector
to gives you the scores back.
So what are the best features to be included
and the best scores for each feature
as it will print it on the screen
once you go through the example.
So here's our library is hosted
on the clarify health part feature selector.
It has adding other ranking.
Our future goal is
to basically add few more ranking algorithm
to the library, right now we just sorted by the scores.
But there are other ways to do it.
We have to convert as you saw,
these are simple functions to you just pass
in the data frame and parameter.
Our goal is to convert this thing
into a pipeline transformer.
So it can be incorporated
in our auto ML pipeline.
So it goes through the transformation phase,
the way we'd use the pipeline.transform API
to seamlessly integrate into our system.
So don't forget to rate the session.
Thank you for listening to us.
Thank you Imran.
And thank you everyone.