Welcome everybody. This is Mr. Fugu and you're watching Mr. Fugu Data Science.
Today, we will do a tutorial for parsing personal data: such as personal profiles using
regular expressions (regex). Remember, all the code will be provided on my Github which
is Mr. Fugu Data Science. If you like this and would like to see any of my other
videos; feel free at Mr. Fugu Data Science. Don't forget if you SUBscribe
turn on your notification bell. Now, let's get started.
The regular expression module documentation, is here. I provided the
link so you can get kind of a more clear and in-depth knowledge of what's going
on. As well as, these two useful tables with tips and tricks. So you can use as
reference material. Which are adapted from the online documentation. We will
need the: import regular expressions, pandas, faker, as well as date/time. I
created 500 fake user profiles. Where I used a random seed. Enabling me to
replicate these data. Now, I converted it to a data frame. So we can get an idea of
what's going on. The last video we use this data and we parsed addresses. Today,
we will be parsing birthday, email address, name, username, website as well as
job title, company, social security number. I had to create fake data because, you
can't exactly put real data online. Now, I extracted the columns that I needed to
use.
Look at that, this is all the relevant information and just described. Then we
take that data and we need to change the formatting for the birthday. And you say
well what's wrong with the birthday format? Well, oh I took it out. Okay, so we
look at real quick. So here's the birthday format
well, it's correct notation in most countries. Where I live, we have a
slightly different notation. Which isn't wrong it's just different. So I changed
it into what I'm more familiar with. Which is month/day/year. and we used the
method of dot s T r F time ( .strftime() ) that allows me to change the string formatting. So
after doing that. I had to create a new column into my data frame. Where I just
adjusted the dates, for the birthdays. Then, I took my data frame; I took all of
the values and converted into a list. From there, I wanted to flatten
everything out. Iterate through my new list, data frame. Well, what does that look
like? Okay, well our list that we're taking
is this. Nope I forgot. Because, it's a list not a data frame. So here's what
our data looks like. So we have this is the first person right here. Occupation,
person's name, the websites they're affiliated with, user name. I don't
remember the difference between these, and then their email, and birthday. Okay,
fine. So we're creating our flat list. Where we're separating by, we're joining
everything based joining all the strings based on a comma. And we're doing that
for each one of these. But, I had to separate this because it's nested and
it'll give you an error. And that's your webpages at that index. So after we
create that, from there we could say what does that flat look like
anyway? Here's our flat list. So what I'm doing is I'm taking everything
let's inside of this, taking everything let's inside of this and flattening it
out. So it's just one big old string. I'm doing that because, I want to just
pretend that if we had data that we found from somewhere. and it contained
all of our user information and there's one string of character separated by
commas. How do we parse that data? All right, so I'm trying to break this data
down and make it more user friendly for if we're in the real world and found
this or extracted this from somewhere. So here's our first our first user that we
took and we made into a string. and so forth right. So we take our flattened
users and then we have to iterate through this. And what's going on with
that? Well, you are taking this and you're joining everything together. and here's
the order let us going. So now we have the websites for the person we have
there occupation, their birthday, their email,
their name, their username, social, I don't remember what this is, maybe company name.
And it has that for all the people. We're dealing with okay perfect. So we move on
from there; and here's just an example of what one of the entries looks like. I
want to start parsing this data. So I take one of the small entries from above
and I extracted it, and I also added a few caveats. Where I said let me just
make up some website without a HTTP one that has just HTTP without the S.
And then just some John Joyce and I just threw that in there. Because, what I
wanted to do is check: are all my regular expressions only evaluating email
addresses that occur at the beginning of a sentence, at the beginning of a string
or not? That's why I threw some stuff at the end. So then I said okay what happens
if I just use this so if I just use this it looks correct. But, it is not. and let
me show you. So this looks correct where I'm taking an HTTP
and I have the question mark, and said well it could have HTTP or could have
HTTP and something else before the colon. and then I took this slash capital S and
I want to find at least one of these. What's this capital S doing? look for
everything except for, except for a space I think is what's going on. and if you
notice: you say oh, okay. I got all of these webpages. But, oh well they
and I got this science junk, got some person named John. But, I got these other
I'm like aw man that's close. But, that's not exactly what I want. So then
the next thing: I say okay well this look for the HTTP or the HTTP okay cool. So I
throw that into a character set and I say well is it HTTP or HTTPS ? Then
it's optional for me to have the literal forward slashes here. and those are
literal and their options. and then I said okay this look for at least one of
these or is also look forward and find us a WWW dot and that's also optional. and
then I put the or operator. and said okay and we don't have to have the WWW dot and
lets look for that at least one time. Then from there anything what happens
after that can be an alphanumeric as well as a hyphen, underscore a period or a
forward slash. Then we were looking for at least one of those and
then after that we can have our lowercase a to Z. Okay, that's fine .so we
can look at what that looks like real quick. and there we are
so we have our HTTP, HTTPS. We have it with the forward slashes. But it does not have W
WW dot. This one does, this one does not. Okay, perfect!
We know we got something going on. We're on a roll. Now this find just our WWW dot friend .
So this look for something, we're forward-looking here. Where
we're looking for the WWW dots. and then we're trying to say okay this look for
everything except a new line. and we we don't need it to be there or not
for this. And then we got our trusty old find it at least once. and then we
look for the rest of the webpage. With this so you got your good old WWW, get
what the heck is going on here? get WWW
all righty... why did that not work? QQ that should
work. Ah, hold on here. Let's verify this then.
Okay, that does work. So I'm missing something here. That, I forgot. is this Plus
this? Always go and backtrack to kind of figure out, what's going on. What
you missed. Oh hold on here.
okay what is going on here let's fix this thing. Okay, that one works.
Okay perfect. There now I got this to work. All right lets look for
our WWW dot. And it doesn't need to occur. Look forward at least one time, and then
find the rest of the address, perfect. And then we just took the above two put
them together: separated them with an or operator, and that's how that works.
Good we're on a roll. Here's a printout of that. Then I just
verified it, against my other data to make sure the data from above is
actually working. Then we look at our social security numbers, and I took my
little toy example: where I just took an actual one, the first actual entry that
happened in our list. And then I added an extra fake social security number into
that. To see if I'm just looking at the beginning of my beginning of my string
or not. So we're saying this look for all digits.
Where we're having a sequence of three character set. Where we look for a
separation by a hyphen, then two digits, a hyphen, and then four digits. Then we say
perfect. So I found this one and the one at the end. So that was pretty easy. Then
we have the email address. The email address has a lot going on. These are
actually quite hard to parse and each situation is gonna be different. And if
you look online through stack overflow you're gonna find out. So here, we have
the option of having lowercase, uppercase letters, we could have a hyphen, we could have
an underscore, we could have a period, I put a plus. And then we're looking for,
you could also have digits. and then its separated by a at sign/symbol. and
from there you can have your A to Z capital in lowercase. and then you could
have a dash, in there in a period. and then also you can add this in there too.
Because, this can occur at the end. You can look at what your data looks like and
split everything into individual strings. And that's what I did here to get an
idea of how the layout is for what's coming up next.
Then I did similar to the social security. Now, our birthdays: where you can
have one or two digits here, that's represented with this then an actual
literal separator, of your forward slash. Then your one or two characters again or
digits. Then your literal, and then your four digits. So that was easy to find.
Then from there you said, okay I have everything. Now, I still have to find
these last bits. If you were looking through this. Here's our string that
we're parsing through. That needs to be split. Which is done above. and we found
everything except, for your personal name, your user name, your company name, and then
this will be your occupation title. That's the last four things we're
looking for. So you have to do an if statement and take everything into
account. And then anything that's left over ends up being this. It works when I
separate it and look at one entry within my larger list of lists. But, when I tried
to do the same thing and iterate through the complete list. I was having issues
with it not working. So that's a note, for anyone who tries to use this. So then
from here: if you're able to get to this point. You could parse this out
really easily because it's already string split. As long as you
reference from the original data what each column means. And then you could
just pull these out, and do indexing. From there, I decided to loop through our
larger list . Which is this whole thing, instead of just one entry. and find the
web page, find the birthday, find the email, social security number,.
The only thing that changed for the coding: is now we're looking for name, and
we're looking for occupation. and I set these up because I was having issues
getting this to work on the whole data set. So from there all of this codes the same
from above just expand it to the full data. We can look at the names and see
how this worked out so it looks like we're close on the names. But, if you
notice something there's some discrepancies. For instance, this found
the person's actual name. But, it also found the company that they worked for.
Because this capitalization. So you'd have to do further parsing on this to
get all the names of the people. Such as the occupations. There are issues. It
could find part it could find this which is an occupation. But, then this is
somebody's company name. Here we are. We're having the same type of issue. This
finds the person's occupation but ,what happens if you have an occupation let's
just one word, instead of two words which are capitalized? So I broke down what,
what this looks like from above. So you can get an idea of the printout to make
sure everything's working correctly. So all the lines here are working correctly.
Except this one needs further parsing, and so does occupation. And the
occupation needs a little more tweaking on the regular expression. I wanted to
break down here one of the examples the first example so you can get an idea of
what's really going on here. Where where there's some hiccups. So for instance
psychotherapist. One word starts with capitalization. But, you have this
scientist, the scientist is actually two words. But, it was split here because the
way I set up the regular expression. Because, I was only looking for a
grouping, let's starting with any type of capital or lowercase letters. I'm going
to cut off somebody's company and then their personal name. So there's
tweaking that needs to be done to that to make this better. That'll be the
end of this video. But, I hope that this was just a primer to get you started. To get
you thinking about how you can parse personal data.
I hope that this provided some utility and you Like this video. So throw me a Like if
possible. and SUBscribe turn on your notification bell if you
subscribe. So you can see the new content coming out a few times per
week. Thank you for watching and see you next video. Bye