welcome back to the data science with MATLAB video series in this video we'll
work on predictive modeling or machine learning alright so we went back and
performed our analysis over all of the years we've done our pre-processing and
we've got a good subset of data to continue to work with now we want to do
a prediction of the damage costs based on the storm events and the other
variables that we have in our table that we've been working with so we'll need to
do a couple of steps first we'll explore different types of ways to approach the
problem then talk about data preparation how to train and test the model and then
sort of what to do with it in the next step so in this case we have a lot of
ways you know just generally speaking with predictive modeling there are many
many ways to approach it depending on your goals and the situation your your
data situation so you know for example you could do curve fitting
classification regression neural networks deep learning all of these
kinds of things and what's nice about MATLAB used in these techniques in
MATLAB is that oftentimes we have apps that'll help you really explore and try
a bunch of different models as you're going through this these tasks and so in
this case trying to predict our damage cost and so that's a numeric prediction
so I'll use regression and before then we want to make sure that we're in we
get the data in the right orientation for regression or for machine learning
so let's take a look at the data that we have just as a reminder so we have our
location the state the time information the type of event the storm duration and
the damage costs and so we want to be able to for our prediction we need to
capture all of the information that might be useful to predict and so we
want to predict based on time but not necessarily in a time series way we just
want to represent the time information so we can just split up the year month
day and hours and use that in the model to see
how those are influencing the predict or the costs so for example you know a
blizzard and July likely is more costly and so we'll use that information in the
model another thing that we want to do is to remove the infrequent classes and
so in this case we'll use we'll remove anything that has less than 200 events
so whenever we're building models like this we also want to partition the data
so that we have a data set held out for testing so we'll split up our data into
training and testing sets and 70/30 split so now I'm gonna train our model
and we use an app and so here I'm just I opened the regression learner and
we'll select the data so I'm just going to select the training data set that we
created and I want to select the response so that's what I'm trying to
predict which is our damage total and then I can select or deselect the data I
don't need so for example I want I don't need the property and crop damage or
latitude longitude because I'm representing that data in a different
way with regression we've got a bunch of
different models and so if you have some indication of what model you might want
to use you could select that or you know maybe you know that it's linear you
could select all all linear in this case I'll just select all of them why not
depending on how many cores you have available you can use parallel and
that'll take advantage train many as many models as you can at once and then
you can take a great look at the visualizations you know even while it's
training to explore you know see some of the patterns perhaps and then there are
some nice visualizations once the models have trained to see to validate the
models so for example you can look at residuals and you know examine the
predictions versus the actual data and so the visualizations also help with
this you know for example some of these categories have a lot of outliers
like our hurricanes and you know it's it's able to predict the or the lower
values much better than the higher values and so these are the kinds of
things that we want to look at in the app and you know take advantage of by
training all of these different models so once we're happy with a model we can
actually export it so we can start to use it for predictions later and not
only can we export the model but we can generate the code and that way we can
repeat the same exact steps that we went through but using a function instead of
you know going through using the app every time you can very quickly use the
function after this and also learn how to adapt the model yourself I exported
the model from the app and now I'm going to use it in our prediction so when you
export the model from the app it's actually inside of a struct and it
contains information about the model and even a function to help you predict
using that model taking the same steps and the same variables as before the app
actually held out data for testing for training and validation we also did that
before training the model and so now we want to test the model on our test data
that we you know had not passed into the app and so now I'll predict based on the
test data and I can use a histogram of the errors or how far away the model
predicted from the real data whenever we deploy this model and we put it into our
app we're gonna be getting the information real time so we'll type in
the location and the type of event but we basically just want to make sure we
have the same steps followed and the data is in the same organization or the
same types that we originally trained the model and so it's you know it would
be very expensive to have a tornado right now in Massachusetts thanks for
watching in the next video we'll look at scaling up our calculations to Big Data