Welcome everybody, your watching mr fugu data science.
Today, were going to parse the indeed job postings qualifications section. Using natural
language processing and if time allows create a word cloud depending on the length
of this video. I do need to bring to your attention:one thing that''s quite problematic
and I don't want to throw off any of my viewers and subscribers. I have an imposter who took
my likeness and just put in my name as their
channel now. But, if you pay attention to the top, this is the same person I made a
video about the other day. My little troll. So please be aware of this. Understand this
is not me. This is me, you have to pay attention to the url. And the only way you see the exact
url: you need to go on the about part. This very important because, this person harassed
me, made fake accounts and told me they would dislike all of my videos. So, please be aware
of this person. I digress. You need a lot of imports today, so I really suggest going
on my Github and getting that. As always it will be in the link and description below.
Feel free to pause the video if you would like to jot this down real quick if you have
interest. Because, there's quite a lot going on. Now, this file comes from our Indeed web
scraping that I did recently. And of particular interests is this right here. This will be
showed in a second for one entry. Our skills match is really interesting because I had
a large, or decent size list of skills that I matched against this text. And each one
of these lists are these particular skills that were relevant that I found were matching
my skills list. And there would be a word cloud for that as well as a second word cloud.
So the first entry will look like this, form this company and it has just a bunch of text.
And were just trying to figure out in the end what are common between all of the text.
I put each of the 225 rows into one large, you could call it a corpus for evaluation.
Here are the buzzwords that I used if you are interested, to evaluate against to make
that skills column. This can be expanded on or customized for your on interests or your
own skillset depending on how you want to do this, but that's available. Now, to get
a baseline let's figure out a few things. I was interested in the number of words for
each row. Because what were doing next, is we need to start breaking this up. And there's
kind of a workflow when doing natural language processing, alright. The first thing when
I evaluated I noticed something interesting. When I did my word cloud I didn't pay attention
and I noted that you need to take out anything that has a url. This right here is going to
take out anything with http and anything after that so it includes the https. So your good
to go. And what I wanted to do is figure out really: ok, is this working and how much of
my data have it. So I found yes; it really is working. Now, we need to actually extract
and pull this from our information. I had to throw in an if statement to work with my
pattern. Then find everything that's within the text for each entry. And then if you don't
have a url: I just chuck this in so we could keep the formatting aligned with the rows.
Now, if i'm getting rid of this I could just use a regular expression and substitute. So
I still am finding the pattern that I am looking for but, instead when I find that pattern
just throw in an empty string with a space. And then I made sure that my lines matched
up. When your trying to recheck this, your basically running the same thing. But, your
throwing out the else statement. Because, if I throw in the else statement here its
appending the rest of the data. But, really i'm just trying to evaluate how many rows
of the pattern that are still left in this data to make sure I didn't make a mistake.
And if you see it: length of zero. So I didn't make a mistake here. We're good to go. Now,
comparing the length with and without the hyperlinks went like this. Because, I was
curious for instance, like this entry. As a sanity check, what does it look like? Its
actually working yes or no. So that's what I did here so you can get a visual representation.
Get a little bit of dialogue, you know. Just like this right here. Now, the next thing
we want to do, is say alright we got this stripped data. If we wanna throw it in we
can enlarge our data frame. Its unnecessary you could just keep it as a list for the next
part that we're going to do. If you notice this right here. When I created the original
file we're reading in today from the web scraping.I think I used an empty string instead of
a space, is why your going to have some of these kind of issues like this. That's just
something to be aware of. Let's remove some punctuation and digits. Throw everything into
lowercase, get rid of the string punctuation as well as any digits and join everything
together with a string join. And append it into its own individual list for each entry.
Ok, that's whats going on here. Now, the next thing that you have to look at are stop words,
single characters and useless stuff if you can find it or If you have an idea what it
really is. I kept 2 and 3 character words because the 2 letter abbreviations for machine
learning, artificial intelligence and deep learning are used quite frequently. So I have
to take only 2 and up instead of 3 or 4 and up. Because, you want to preserve some of
those. That's whats going on here with this list comprehension. And from there you have
to notice something here. I have a double bracket, because without the double bracket
its not going to enclose this in the formatting here. Its just going to print it off as strings
separated by commas. Thats one thing you gotta pay attention to and your just chucking in
your stop words that's from nltk that you imported in. If you want to see what the stop
words really looks like, if you haven't done this before. This is basically what your looking
at. Here's a bunch of the stop words. The same thing if you didn't pay attention above.
Here's what your punctuation looks like that your parsing out, ok. We gotta see as a comparison
what's left from what were cleaning. And if you notice its about 10 times smaller. In
general your gonna have a lot of stop words and just useless things. Since, I removed
digits, stop words, urls and things like this it started progressively getting smaller very
quickly. The next thing you can do and it depends on your situation. For instance, I
thought here it would be useful to do lemmatizing, you could also do stemming. But, be aware
of some of the hurdles or problems that you can have with that. I suggest you look at
differences in stemming and lemmatizing. Stemming is cutting the ends off of words. Whereas
lemmatizing deals with actually a rule based approach. And there are different types of
lemmatizers and stemmers. Now, what I wanted to do since this is a lists of lists. I needed
to get inside of all of that. And I wanted to preserve the index position, that's what
I is. And take my actual word that is J and throw that into a default dictionary that
I can use for what we will have in a second.This is the first entry in the first row for
all the words that are separated. That's whats going on. And I have it for each on of the
rows 225. But, really, all I need was the values of it. Then after we get the values,
I want to check that the length of this didn't change. Which it should not, because its keeping
the same word. But, its just affecting the ending, ok. I printed this off so you can
understand what is going on with that. I threw it into the data frame so now we can have
this. Which we will use next. Let's back track and think about what we just did as a workflow.
I had to figure out one: is there anything odd that I have to take care of? The first
odd thing that I found which was the hyperlinks that I needed to take out because their just
going to cause more confusion and more useless words when split. Second, you start looking
for stop words. If you have a text, you want to split that into tokens, tokenizing. You
can split that into either sentences or individual words. Usually, what you do. Then you will
decide:do I need to take out digits? Do I need to take out individual words that were
not inside the stop words. So your making a separate list or dictionary for you to do
further processing. We have something that's interesting. The lemmatizer, you would think
would adjust for something like this. But, that's not the case. Because anything with
scien, comes back. And allows me to see that it was not changed. And I didn't realize this
until I ran one of the word clouds to understand that there's actually a problem here. And
this is something that you can come back to and rectify I suggest or deal with it and
explain the discrepancy, if need be. We needed to convert our lists of lists into a single
large string. So I had to iterate inside of this because its a lists of lists. That's
why there's two loops. And then I want to come back and throw this into a data frame.
You can also do a direct pandas function, I thinks thats what it is, explode. But, don't
quote me on that. And then from there I took that one big list of words here and converted
it into one large string that we're using for our processing. Then I figured, how would
I start looking at the frequency of words to start seeing there's something in particular
that may throw off my words cloud later. Because, that's what your working with.Your basically
using a words frequency to dictate how large a word will be in your plot is how how can
think of it. The word data, is one of the biggest words in there and science, from data
science or maybe part of some education requirement from some scientific background. Its hard
to tell unless you look independently at each posting. But, here you could use this as a
tool to say hey, I sorted this out let's look at the really strong, highly captivating word
frequencies and see if there is anything in particular that is useless and I can chuck
that out in further processing, ok. But, I hope this brought utility to someone. Please
like, share and subscribe. And if you subscribe turn on that notification bell. And remember
do not go to this persons site who is imposter. They're mad at me, because I made the video.
But, that's how it goes man. So please change that profile picture. And don't try to be
me. Your not me, and don't try to be me. Your not me. Your not gonna be me. Your not me.
So please don't try to throw anymore animosity. Because i am just gonna throw shade. Don't
be a troll. Your always gonna have haters trying to take you down no matter what you
do. So keep on trucking. There's always gonna be trolls, so have a good day mr no body.
Because your not me. This is me. Have a good one. Bye