welcome back to the data science with
MATLAB video series this video will
focus on pre-processing the data which
is a really important step in the data
science workflow now that we've brought
in the data brought in a subset of the
data we want to do some pre-processing
and so there are lots and lots of ways
that we want to approach this problem
and I'm actually just gonna go through
all of them because this is what we work
with in data science data are just messy
they're all different types so we're
gonna talk a little bit about using the
best MATLAB data type for the job how to
access the data pre-processing with
different kinds of data types and then
one of the biggest headaches for me and
many of us are working with missing data
and so you know we'll spend some time
talking about that and then other common
situations like outliers merging data
resampling and all those kinds of things
that will we'll dig into
so we're gonna work with our storm event
data and then we're also going to merge
this in with some current weather data
just to explore a little bit and make
sure that our weather data are on the
right times so let's start by examining
our data and so just to kind of remind
us what we're dealing with and sure
enough we've got numeric data we've got
our categorical data like our state
event types we've got x so the beginning
and end time of each storm then we have
our damage costs and these are currently
actually strings so you know it says 2k
and so that's representing thousands so
we'll have to figure out how to deal
with that we also have the source so you
know where the data were recorded from
the latitude longitude which we'll also
need to convert to numeric and then we
have this event narrative which is
really interesting it's a text a free
text field that someone just described
the storm
and so in our case also access data in
more complicated ways so for example if
we want to just select data from one
location we can do that by using logic
so if we want to just see how many data
points are from Alaska
we can just say data state equals Alaska
and so we can also use this to access
the data so I can use that logical
expression to represent the rows that I
want to bring from the data set
okay so that's how I would do this for
one state say in this case I just want
to use data from certain sources from
the US government and so I can also
select that very similarly to how we
just did by representing the rows as our
logical expression and then selecting
all of the columns of data and so since
these are also in tables I have some
nice functions I can use to reorder and
reorganize the data as I see fit or
remove the variables I don't need
anymore like the event ID in a source
and so I moved the variables the time
variables in front of the other ones so
I've been talking about tables quite a
lot let's actually talk a bit more about
how to represent data in MATLAB
generally speaking there are sort of our
traditional data types which are numeric
char data sort of your low-level kind of
data types and then over the past couple
of years we've introduced a lot of
really nice data types for data science
kind of situations so you'll see these
in the example like categoricals date
times durations our table and time table
can actually contain any of these
variables and so that's what what's
different about the tables that we've
been using so far you can contain a
number of different data types in that
table similar to a spreadsheet and so
all these also could be made tall and so
we'll talk about that a little bit later
to where you can you know treat the data
it's actually out of memory just as a
typical MATLAB variable and so that kind
of gives us you know the overview of the
different data types let's actually now
talk about some of the benefits of using
these data types so we'll start by doing
our time series pre-processing all right
so let's take another look at our table
again and so again we have our begin
times and end times and if we wanted to
do you know pre-processing with those
times maybe want to change the dates we
can do
that using D time variables so for
example if we take a look at our
beginning timestamp this is our you know
time of year and then time of day and in
this case it's not doesn't really
indicate which one is the year its I
probably can assume it's not the future
but we might want to just update the
format to our liking so it uses the
default from the file but we can adjust
this as we like and so now it's a little
bit more readable and now I can actually
determine the duration of each storm and
so if you imagine if we're trying to
predict the damage costs or the impact
of these storms a very long tornado that
lasts for three hours is likely more
costly than a ten minute tornado so this
is a very important variable for us in
our exploration and so now I've got the
storm duration and we can take a quick
look at this we can just use our typical
stats functions like Max mean those
kinds of things to explore the time so
for example the max duration was 743
hours which turns out to be about a
month and you know the mean is about 17
hours if we look at a histogram this
gives us a better idea of the
distribution and you know sure enough
these are all sort of pretty low but
then we have a couple out here that are
you know around a month but let's
actually look at some of the top it's
about the top 10 so we were looking at
the longest storms lasting for a whole
month
let's actually pick out the top 10 most
longest longest storms and check these
out and so in this case these are all
droughts and so that kind of makes a
little bit more sense than a hurricane
lasting for a month alright so if we
want to do a bit more pre-processing
with our time we can create a time table
and so it's very similar to a table
which we've been working with but it
actually uses the time stamps as the
rows of the table and so you can access
the data with the time and you can do a
lot of
nice pre-processing steps that are sort
of headache sometimes when you're
working with time-series tables and time
tables also contain properties that can
help contain metadata and so with these
applications you often have a lot of
information about the variables that you
can capture here and so you know we have
our variable names we have the you know
time information and then any custom
properties that you want to add yourself
so our time dimension is called begin
time stamp it might be easier if I just
call it time and so let's also take a
look at the range so I want to see you
know make sure that the time range is
correct and so sure enough this is from
January 2016 to December 2017 let's now
take a look at our categorical data and
so a categorical data it's more it's
like a label or a repeated label like in
this case we have the US state or the
event type and if you use the actual
categorical data type in MATLAB you have
a lot of really nice functionality that
comes with it that will help you work
with these kind of variables and so
first off I can just get a quick summary
there's me an idea of what sort of
categories I have and how many and so I
can see already that I'm gonna need to
do some pre-processing you know I
probably want to remove things over
bodies of water and then for my event
types some of these could be combined
you know for example you know maybe
heavy rain and rain could be considered
the same category I also have a very few
of these so you know we might want to
remove those altogether when we're doing
predictions so we can also take a quick
look at this with a histogram I've
created a function to make a bigger
figure a tall figure so that I can see
all of this at once and so you know as
you can see there are lots of
thunderstorm winds high winds
lots of tornadoes winter storms and so
you can get a quick idea of the
distribution of these events so as I
mentioned we probably want to combine
some of these just to make a little bit
more sense and so maybe with wind we
could confine wind and high wind to just
be all win and I'm gonna continue this
over and over again so I've created a
live function
that goes through the same process for a
multi multiple other variables so for
example you know snow heavy snow flood
heavy floods those kinds of things will
all be combined and so I'll go back here
use my live function and check out the
new categories so sure enough this is a
much more manageable list of categories
and we still have good interesting ones
like dust storms and debris flow alright
so that takes care of our categorical
data and we also have some textual data
that we need to work work with so some
of it is textual by Nature like our
event narratives so you know the things
the free text that somebody actually
wrote so that's appropriate to keep as a
string data type in MATLAB the other
ones in our dataset are not appropriate
as strings so we're gonna need to
convert these and for me I always if I
don't know what the data type should be
or I'm not sure what it looks like I'll
just bring it in as a string so I can go
ahead and pre process and convert later
just because it's very friendly to do
that so first I'll make sure all of the
event narratives are lowercase just so I
can more easily you know
analyze those and then I have another
function to do our string pre-processing
that converts our data types and so for
example it's just going through and
converting our latitude and longitude
directly just to double precision and
our damage cost variables it's detecting
where it says M or K for millions or
thousands and you know changing the
units based on that and so now we'll run
that function and we'll check out our
results now we have a much more
manageable data set and our numbers are
actually numbers and our text is text
all right so now that everything is in
this the right data type we have a lot
of missing data so we want to figure out
how to approach this and so sometimes we
want to remove these sometimes we want
to fill them you have lots of options in
MATLAB so you know we could standardize
we can
check them remove or fill them with a
bunch of different options and so in
this case let's just take a quick look
to see how much I actually have and so
if we we can use is missing and that
lets me know that you know the some of
them have no missing data some of them
have you know 13,000 missing data points
but first things first we want to remove
rows that have all missing data and so
if it's all missing except for the time
that's not helpful to us another thing
that we need to do is add our damage
costs but sometimes there are missing
data in one of those variables and so I
can tell the function to omit Nan's so
you know just ignore the missing data as
you're calculating all right so our
event data is looking good and now we
also want to compare this to current
weather data and so we actually have at
our math works headquarters in Natick
Massachusetts we have sensors on our
building that give us weather data we
use things speak to store the data from
the sensors and I can use things speak
read to bring that into MATLAB through
things speak read I can choose the date
range so I want to make sure I'm
choosing a range from the study and it's
bringing back you know typical
meteorological information like the wind
direction wind speed temperature
humidity all that kind of stuff
and we can take a quick visual a quick
visual look at this with a stacked plot
whenever you pass a time table it'll
actually show you all of the variables
at once and so as you can see you know
this is also a bit messy you know there
are some outliers we're probably going
to want to do some smoothing you know so
it's a little bit different types of
pre-processing so when it comes to
outliers it's very similar to what to
our approach with missing data so I can
detect them with is outlier and then I
can decide to remove or fill them with a
bunch of different methods that the
function provides so first let's check
to see if I have outliers and I can
choose the method to detect the outlier
so in this case I'm doing a median and
so you can see some have multiple you
know some have very few I know that with
the light intensity data I want to just
go ahead and fill those
altogether another thing with sensor
data is that you know sometimes you
might have you know hourly or you know
data on the minute or on the second and
we want to just make sure this is
consistent throughout the whole data set
it's much easier in our analysis that
they're on the same time and so we can
resample or read time and in this case
I'm gonna use linear interpolation to
put all of the data on the minute and
sure enough these are all on the same
time now and I can carry on with some of
my further analysis and so you know we
notice there's a lot of up and down in
doing my analysis you know I can I can
just work with a smooth to set from here
on out so the last thing I want to do is
to synchronize the data and so you know
I mentioned that we have all of this
event data and we only have our sensor
data for you know obviously
Massachusetts where the sensors are
located and so we can put these data
sets together but we want to make sure
that we're being careful about the times
and so if we use synchronize we can
actually choose what time we want in
this case I'm saying first so that's
gonna put everything on the time of the
weather data and then I can see that you
know February 5th
there was a winter weather warning on
the 8th there was a blizzard and so if I
look at a plot I can see that you know
these are some of the points where you
know the blizzard happened and there was
wind that was likely related winter
weather you know all kind of related and
nice to see you on the visual thanks for
watching in the next video we'll look at
analyzing our data