welcome back to the data science with MATLAB video series this video will
focus on pre-processing the data which is a really important step in the data
science workflow now that we've brought in the data brought in a subset of the
data we want to do some pre-processing and so there are lots and lots of ways
that we want to approach this problem and or I'm actually just going to go
through all of them because this is what we work with in data science data are
just messy they're all different types so we're gonna talk a little bit about
using the best MATLAB data type for the job how to access the data
pre-processing with different kinds of data types and then one of the biggest
headaches for me and many of us are working with missing data and so you
know we'll spend some time talking about that and then other common situations
like outliers merging data resampling and all those kinds of things that will
we'll dig into so we're gonna work with our storm event
data and then we're also going to merge this in with some current weather data
just to explore a little bit and make sure that our weather data are on the
right times so let's start by examining our data and so just to kind of remind
us what we're dealing with and sure enough we've got numeric data we've got
our categorical data like our state event types we've got times so the
beginning and end time of each storm then we have our damage costs and these
are currently actually strings so you know it says 2k and so that's
representing thousands so we'll have to figure out how to deal with that we also
have the source so you know where the data were recorded from the latitude
longitude which we'll also need to convert to numeric and then we have this
event narrative which is really interesting it's a text a free text
field that someone just described the storm
and so in our case also access data in more complicated ways so for example if
we want to just select data from one location we can do that by using logic
so if we want to just see how many data points are from Alaska
we can just say data state equals Alaska and so we can also use this to access
the data so I can use that logical expression to represent the rows that I
want to bring from the data set
okay so that's how I would do this for one state say in this case I just want
to use data from certain sources from the US government and so I can also
select that very similarly to how we just did by representing the rows as our
logical expression and then selecting all of the columns of data and so since
these are also in tables I have some nice functions I can use to reorder and
reorganize the data as I see fit or remove the variables I don't need
anymore like the event ID in a source and so I moved the variables the time
variables in front of the other ones so I've been talking about tables quite a
lot let's actually talk a bit more about how to represent data in MATLAB
generally speaking there are sort of our traditional data types which are numeric
char data sort of your low-level kind of data types and then over the past couple
of years we've introduced a lot of really nice data types for data science
kind of situations so you'll see these in the example like categoricals date
times durations our table and time table can actually contain any of these
variables and so that's what what's different about the tables that we've
been using so far you can contain a number of different data types in that
table similar to a spreadsheet and so all these also could be made tall and so
we'll talk about that a little bit later to where you can you know treat the data
it's actually out of memory just as a typical MATLAB variable and so that kind
of gives us you know the overview of the different data types let's actually now
talk about some of the benefits of using these data types so we'll start by doing
our time series pre-processing all right so let's take another look at our table
again and so again we have our begin times and end times and if we wanted to
do you know pre-processing with those times maybe we want to change the dates
we can do all that using D time variables so for
example if we take a look at our beginning timestamp this is our you know
time of year and then time of day and in this case it's not doesn't really
indicate which one is the year its I probably can assume it's not the future
but we might want to just update the format to our liking so it uses the
default from the file but we can adjust this as we like and so now it's a little
bit more readable and now I can actually determine the duration of each storm and
so if you imagine if we're trying to predict the damage costs or the impact
of these storms a very long tornado that lasts for three hours is likely more
costly than a ten minute tornado so this is a very important variable for us in
our exploration and so now I've got the storm duration and we can take a quick
look at this we can just use our typical stats functions like Max mean those
kinds of things to explore the time so for example the max duration was 743
hours which turns out to be about a month and you know the mean is about 17
hours if we look at a histogram this gives us a better idea of the
distribution and you know sure enough these are all sort of pretty low but
then we have a couple out here that are you know around a month but let's
actually look at some of the top it's about the top 10 so we were looking at
the longest storms lasting for a whole month
let's actually pick out the top 10 most longest longest storms and check these
out and so in this case these are all droughts and so that kind of makes a
little bit more sense than a hurricane lasting for a month alright so if we
want to do a bit more pre-processing with our time we can create a time table
and so it's very similar to a table which we've been working with but it
actually uses the time stamps as the rows of the table and so you can access
the data with the time and you can do a lot of
nice pre-processing steps that are sort of headache sometimes when you're
working with time-series tables and time tables also contain properties that can
help contain metadata and so with these applications you often have a lot of
information about the variables that you can capture here and so you know we have
our variable names we have the you know time information and then any custom
properties that you want to add yourself so our time dimension is called begin
time stamp it might be easier if I just call it time and so let's also take a
look at the range so I want to see you know make sure that the time range is
correct and so sure enough this is from January 2016 to December 2017 let's now
take a look at our categorical data and so categorical data it's more it's like
a label or a repeated label like in this case we have the US state or the event
type and if you use the actual categorical data type in MATLAB you have
a lot of really nice functionality that comes with it that will help you work
with these kind of variables and so first off I can just get a quick summary
it's me an idea of what sort of categories I have and how many and so I
can see already that I'm gonna need to do some pre-processing you know I
probably want to remove things over bodies of water and then for my event
types some of these could be combined you know for example you know maybe
heavy rain and rain could be considered the same category I also have a very few
of these so you know we might want to remove those altogether when we're doing
predictions so we can also take a quick look at this with a histogram I've
created a function to make a bigger figure a tall figure so that I can see
all of this at once and so you know as you can see there are lots of
thunderstorm winds high winds lots of tornadoes winter storms and so
you can get a quick idea of the distribution of these events so as I
mentioned we probably want to combine some of these just to make a little bit
more sense and so maybe with wind we could combine wind and high wind to just
be all win and I'm gonna continue this over and over again so I've created a
live function that goes through the same process for a
multi multiple other variables so for example you know snow heavy snow flood
heavy floods those kinds of things will all be combined and so I'll go back here
use my live function and check out the new categories so sure enough this is a
much more manageable list of categories and we still have good interesting ones
like dust storms and debris flow alright so that takes care of our categorical
data and we also have some textual data that we need to work work with so some
of it is textual by Nature like our event narratives so you know the things
the free text that somebody actually wrote so that's appropriate to keep as a
string data type in MATLAB the other ones in our dataset are not appropriate
as strings so we're going to need to convert these and for me I always if I
don't know what the data type should be or I'm not sure what it looks like I'll
just bring it in as a string so I can go ahead and pre process and convert later
just because it's very friendly to do that so first I'll make sure all of the
event narratives are lowercase just so I can more easily you know analyze those
and then I have another function to do our string pre-processing that converts
our data types and so for example it's just going through and converting our
latitude and longitude directly just to double precision and our damage crossed
variables it's detecting where it says M or K for millions or thousands and you
know changing the units based on that and so now we'll run that function and
we'll check out our results now we have a much more manageable data set and our
numbers are actually numbers and our text is text all right so now that
everything is in this the right data type we have a lot of missing data so we
want to figure out how to approach this and so sometimes we want to remove these
sometimes we want to fill them you have lots of options in MATLAB so you know we
could standardize we can detect them or fill them with a bunch of different
options and so in this case let's just take a quick look to see how much I
actually have and so if we we can use is missing and that lets me know that you
know the some of them have no missing data some of them have you know 13,000
missing data points the first things first we want to remove rows that have
all missing data and so if it's all missing except for the time that's not
helpful to us another thing that we need to do is add our damage costs but
sometimes there are missing data in one of those variables and so I can tell the
function to omit Nan's so you know just ignore the missing data as you're
calculating all right so our event data is looking good but now we also want to
compare this to current weather data and so we actually have at our math works
headquarters in Natick Massachusetts we have sensors on our building that give
us weather data we use things speak to store the data from the sensors and I
can use things peak read to bring that into MATLAB through things speak read I
can choose the date range so I want to make sure I'm choosing a range from the
study and it's bringing back you know typical meteorological information like
the wind direction wind speed temperature humidity all that kind of
stuff and we can take a quick visual a quick visual look at this with a stacked
plot whenever you pass a time table it'll actually show you all of the
variables at once and so as you can see you know this is also a bit messy you
know there are some outliers we're probably going to want to do some
smoothing you know so it's a little bit different types of pre-processing so
when it comes to outliers it's very similar to what to our approach with
missing data so I can detect them with is outlier and then I can decide to
remove or fill them with a bunch of different methods that the function
provides so first let's check to see if I have outliers and I can choose the
method to detect the outlier so in this case I'm doing a median and so you can
see some have multiple you know some have very few I know that with the light
intensity data I want to just go ahead and fill those
altogether another thing with sensor data is that you know sometimes you
might have you know hourly or you know data on the minute or on the second and
we want to just make sure this is consistent throughout the whole data set
it's much easier in our analysis that they're on the same time and so we can
resample or retime and in this case I'm gonna use linear interpolation to put
all of the data on the minute and sure enough these are all on the same time
now and I can carry on with some of my further analysis and so you know we
notice there's a lot of up and down in doing my analysis I know I can I can
just work with a smooth to set from here on out and so the last thing I want to
do is to synchronize the data and so you know I mentioned that we have all of
this event data and we only have our sensor data for you know obviously
Massachusetts where the sensors are located and so we can put these data
sets together but we want to make sure that we're being careful about the times
and so if we use synchronize we can actually choose what time we want in
this case I'm saying first so that's gonna put everything on the time of the
weather data and then I can see that you know February 5th
there was a winter weather warning on the 8th there was a blizzard and so if I
look at a plot I can see that you know these are some of the points where you
know the blizzard happened and there was wind that was likely related winter
weather you know all kind of related and nice to see you on the visual thanks for
watching in the next video we'll look at analyzing our data