welcome to the data science with MATLAB video series in this video we'll focus
on importing the data which is the first step in the data science process in this
case we actually have a bunch of CSV files and I have them in this storm
event data directory as you can see there's a pile of them and they go again
from 1980 to 2017 and so if you're just wanting to explore the data and start
bringing it in you can use the import tool and so this is great for reading
one file and again just kind of looking at what you even have right so you can
interactively explore you can choose the data types that you want to bring it in
as which we'll talk about later so if you want in this case it's going to
bring it in as a table you can choose individual data types or individual
names and so again this helps really explore and choose how to work with some
of this data for example if matlab wasn't able to read it it will replace
it with an an and a great thing is that you can
actually generate code from this so if you do this once and you have many files
like this you could actually generate a function like this and then use that
over and over again to bring in the data so there's also if you're reading just
one file there's a few nice functions for this so for example read table and
so this just brings in the data just like you saw previously with the import
tool and it just reads it into MATLAB for me and so again this is great if you
have one file but in our case we have multiple files spanning over those
different years and we need to figure out there's well they're also organized
in the same exact way and so there's a way that we can do this that's called a
datastore and so a datastore is a great way again you know anytime you have
multiple files or extremely large files or you know databases with a bunch of
data you may not be able to just read it all into memory directly or you may not
want to and so a datastore is a great way to manage that in MATLAB because it
just really points to the data and then you can kind of choose how much data you
want to access you know what types of data you can
control a lot of that and so we'll do that for this example in our storm event
data so I've created a datastore and as you can see it's pointing to a number of
files and it gives me some shows me some of the options that I have and so I
don't really want to take I don't want to think about all of that right now
I want in battle AB to do some of the work for me so I'm just going to get a
quick preview of the data and see what types and what organization MATLAB has
decided based on the files I also want to make sure that all the files that I
want were represented and so I can do this by checking different properties of
the data store and so here I can take a look at all the files listed and sure
enough they go from 1980 to 2017 and so to start out I don't necessarily want to
look at everything right away I just want to look at the most recent data so
I'm just gonna go back and select 2016 and 2017
so that takes care of the files or the dates that I want that I'm interested in
I also want to choose which data properties or which data variables that
I want and other properties about those so there are 51 of these variables again
there's many representations of time I really only need one or two of those and
so I can preview again and make sure I have just the data that I need and take
a quick look so I've got the end locations and time begin time sources
and our event narratives and so as you can see there are many different data
types included here so we want to make sure that we're using the best data
types for the job MATLAB will try to make the best
decision but you also have control of this through the data store itself and
so for example I can create categorical variables for the state and the event
and just ensure that the date variables are using the right types all right and
so now I've got the right data types I've got the right variables selected
and now I want to decide how much data to bring in and how to work with that so
with a data store you have several options there are so we saw a preview
that reads in just the first few lines so you can get just a feel for a quick
feel for the data you can also read a subset of the data so you can just use
read and that will use the read size which I'll show in a second you can also
read all so if you want to just bring everything from the data store into
memory go for it or you could use a tall array I'll actually show this a bit
later where you actually work with the data out of memory and so you just
pretend like it's any old MATLAB variable but it's actually pointing to
that out of memory data so for now I want to just read in a subset so I can
carry on my analysis you know as I usually do and so I want to check the
default read size which we can do in this case it's 20,000 lines and so if I
use read as I'm doing here this just brings in the first 20,000 rows
the second call to read will you know take the next 20 thousand rows and so on
and so that's a great way to just manage you know taking small subsets and doing
analysis that way in this case since I only used I only selected two years I
can actually just read all of that in I know I have enough memory to manage that
and as it turns out there was a hundred and five thousand or so lines so that's
a manageable set and I'll continue through my pre-processing now that I've
read the data thanks for watching in the next video we'll look at pre-processing
the data now that we've imported it