Hi this is Jeff Heaton, welcome to Applications of Deep Neural Networks,
with Washington University. In this video we're going to look at how to handle
files in Python. This gets us ready for doing much more complicated things with
Keras and in TensorFlow. We'll see in particular how to deal with
text files and image files because these are the primary types of files that
we'll be dealing with in this course. For the latest on my course and projects,
click subscribe and the bell next to it to be notified of every new video. Let's
have a look at Python file handling in this course. On neural networks we're
going to deal with essentially three types of files, we're going to deal with
CSV files, image files, and text files. CSV files usually have the .csv extension, you
can think of those as looking like Microsoft Excel files. In fact on most
computers you double-click a CSV file and Excel will pop open. Image files, we'll
deal mostly with PNG or JPEG there's also text files and others. Images are one
of the things that neural networks do particularly well so we'll see a lot of
different images in this in this class. And then text files have the .txt
extension. Those are usually just raw text and that has to do often with
natural language processing other types of files that we'll see. Briefly in this
class are JSON and h5 these files will come from three primary locations your
hard drive. Now this is important if you're using Windows you'll see paths
like this, if you're using a Mac you'll see files like this, most of you I assume
will be using Google CoLab so watch through the tutorial on how to read
those from Google CoLab you'll basically have a path just like this,
like this, it'll be close to Macintosh. CSV files can be read with
Pandas. We will deal with Pandas a lot in the next class. Another thing to note
here that is the way that i'm doing this. I am giving you web addresses for most
of the csv files that will load data HeatonResearch.com that's my own
URL which is unique to this class and I have all the data files there. This way
if you're running in Google collab or Mac or Windows or whatever it will work
just fine. The above command loads Fisher's iris dataset straight from the
Internet. Ok, it's loaded you can display the first five records of that. And this
is a very classic dataset if you haven't seen it before it is basically
four measurements from flowers and it defines the species. Here there's
actually three different species at the very beginning since the file is sorted
you're only seeing the first iris which is sat Setosa. This is a classic
classification dataset where you try to use these values to classify what type
of iris flower it is. Reading an image requires the PIL library but it works
similar to reading CSV files you're basically going to do this and run it
and it will load this image from from this URL. And this is one of the
buildings from Washington University, it doesn't actually look like that anymore
thanks to all the construction but for at least the grass in front of it. But this
is this is a way that you load a JPG. Now when we do some of these exercises we're
going to have lots and lots and lots of images. You may literally harvest the
images this way but you'll want to put them on a folder on your Google Drive.
And I show you how to do that in the introduction to Google CoLab video so
make sure you watch that if you're using if you're using Google CoLab and
believe me for some of the image processing that we'll do in this class
you'll want Google CoLab, because it has a GPU you can also read or stream large
CSV files. What's good about streaming them is you
don't actually load the whole CSV file into memory at once. if you're just
reading a CSV file and calculating statistics on each row you really don't
need to load the whole thing into memory you can simply read each row and process
it. So here I am taking the iris data set I am opening it up for streaming that
means the whole thing is not loaded into memory at once and I'm going to sum each
of those four values that we saw in there. I create an array of four zeros and
NumPy and I keep a count of how many rows I've read and then I run this and
you'll see this is this is essentially the average of each of those four values
I am going to loop over the lines so each each line I basically kept to I
convert that line into. Let's see more of this when we when we deal with NumPy but
this is how you create a numpy array I'm taking locations 0 to 4 only. I don't
want the species the species is the fifth and I convert them into
floating-point because they're integers and because they could potentially come
in as integers or strings or other values. I want them as floating points if
the line is the right length I skip any empty lines. I believe there's an empty
line at the end I sum it up this is a vector summation so this is basically
taking everything that's in that line too so the array of those four values
and I'm adding it to that other array that I have and I keep the count.
Incremented so that's a vector addition that's adding for our numbers just like
this is a vector division this returns a vector or an array back but it divides
each element by count vector mathematics is very useful in this this course,
because we're often dealing with vectors which are a low dimension form of a
tensor. So we we deal anytime that we can perform math across an entire tensor at
once, it's a great optimization really a text file this is the United States
Declaration of the independence this is just a text file that I was able
to find that had a URL, and it was a pure text file you can run it and it
basically prints out the raw text of the Declaration of Independence of the
United States of America. Thank you for watching this video this concludes file
handling and Python in the next video. We're going to look at functions lambdas
and MapReduce this content change is often so subscribe to the channel to
stay up to date on this course and other topics and artificial intelligence.