Okay, let's take a look at an example of applying this procedure
we've just talked about. Now in this case, what I want to
do is work with the open street map data for the city
of Chicago, Illinois in the United States, I chose this because Chicago is
my hometown. Now looked at an example from this data set in
a previous lesson when we were talking about extracting data from XML.
Now, in this example, we're going to take a look at a
small part of the cleaning that we would need to do for this
data set. This is all human-entered data. Or, for the
most part, human-entered data. So, it is especially dirty. What I'd
like to look at here is auditing street types. So,
for example, avenue. One thing we might be interested in doing
with a data set like this is harmonizing all of
the data, so that we have just one way of writing
something that's an avenue. Alternatively, what we might actually be interested
in doing is encoding the fact that this is an avenue
somewhere in the tag set for this particular address. Or, encoding
it in some other way. Okay? So, the type of data
that we're going to be getting in to our auditing routine
is an entire street name. And what we're going to be
extracting is just the street type. And, then, taking a look
at all of the different street types that are mentioned in
the data set that we have. Now the data set that
we're going to work with here Is actually only a fraction of
the complete Chicago Open street map data set. In this case,
we're going to be working with about 20% of the total data
set. The entire data set is almost 2 gigabytes. So for text,
that's a pretty large data set. Okay, back to the code. So,
as I mentioned, we're going to be getting street names, and
what we want to do is recognize types from street names. Now,
I'm going to use Python's regular expression module in order to parse out
the street types. So, let's take a look at the regular expression
that's here. Okay. We're going to match a sequence
of non-white space characters optionally followed by a period.
This is to catch abbreviations like ave or st
as an abbreviation for street, and this match must occur
at the end of the string. And we're going
to use this regular expression right here. And each time
this function audit street type is going to be
parse the street name. We're going to do a match
against that string looking for something like ave or avenue
or street. Okay. Let's take out more holistic look at this
particular piece of code. Now, again, this is just a
small portion of the type of auditing that we would want to
do, so the way I've set this up is the
main routine here is called audit, and what audit is doing
is it's looping through this XML file using the Sax
parser that's part of the element tree module. From Python. Let's
just take a look there. See? And in this case I'm
actually using the C implementation of this particular module. So I'd
encourage you again to go take a look at the documentation
for XML etree, in particular the element tree module. Okay, now
what we're going to be doing here is looping through that
file. We're going to get one tag at a time here.
And, what I'm doing each time is asking the question, is
this a street name? Now, if we were doing a full
auditing parse, we might actually have multiple things that
we're checking here and be generating multiple types of
audit data. In our case, what we're going to
be doing is essentially creating a record of all
street types that we find in this data set.
So, let's go ahead and run this. Okay, and
here's out output. Now, what I did here, was
essentially keep a record of the number of times I
saw all the different street types. And my parser recognizes the
street type from the street name. We can see that we've
got four different forms. Of avenue, the most common one being
the full word avenue and then ave as an abbreviation without a
period, although there were several that had a period. And again
remember this is only 20% of the data type. Okay, the other
thing that's nice about auditing the data this way is I
would have forgotten about street types like court. For example, and I've
actually got three different types of court represented here, and
probably also boulevard. And we can see that that also
is represented. So what we've done here is essentially audit
all of the street types as entered by the human editors
over the open street map data. Now what we would
do with this particular type of data Is then decide,
okay, what type of cleaning is necessary? Do we want
to convert all forms of avenue into the full word avenue.
And, do something similar with all of the other street
types. What type of capitalization do we want? Do we want
something that looks like this? Or, do we want just the
first letter capitalized? And then finally, we can see that there
are lots of things that actually matched our parser, which
are not in-fact, street types. So, we need to be careful
about that, as we're actually doing the cleaning. So that we
don't In fact, introduce further errors into the process. Okay? So,
I'd encourage you to take a look at this
code which is shared with you and make sure you
understand how it works. The next step then, would
be to actually use the audit data. That we have
here, in order to figure out some cleaning tasks,
for street types. And again, as I mentioned, this is
just a small portion of the type of cleaning that
we would have to do for this particular data set.