Today we're going to start with this doing some parsing from Indeed
helping out a viewer. This should work for other parts of the world. And here's what
we're going to get at the end of this video. We're going to turn that data
after web scraping into this data frame or decide to save it. Let's get into it.
Welcome everybody you're watching mr fugu data science. We're going to look at Indeed
web scraping once again. Last video i did was October last year and then one the year before
and now it's June 2022. It appears that they keep changing the way everything's set up
can be a little issue for coding this stuff for web scraping. I want to have an honorable mention
right here for this viewer and recent member for my channel. I'm showing my appreciation and
doing a shout out thank you very much i greatly appreciate it. I assume today's code will be
relevant for other countries. Feel free to use it if there's any discrepancies reach out to me. I do
have a community tab for my channel so feel free to leave a message. Here's the indeed.com
you need to pay attention here if you're in a different country. Because the way that you set
it up can cause you issues with parsing this and what i'll show you next in the actual code
are actually different. Because of discrepancies while i was trying to parse. We're looking at data
analysts today in the region of Chennai in India. So let's update this to get the current postings.
I'm going to sort by the date. Because that's how i actually coded it up. I'm using Chrome.
So just go into more tools and then let's go into developer tools. Not sure where you're at for web
scraping if you've done it before or if you have not. I have two prior videos for indeed if you're
curious. They all have different code for data i'm extracting which may or may not be relevant
for you. I'd suggest checking those out to get ideas of different things to parse. Click this real
quick and look at our first entry and we'll notice that we have this break right here. So to scroll up
and figure out where the outermost portion is that contains each one of our job postings. it's held
within this div class mosaic zone with an id of mosaic zone job cards. That's holding each one
of our entries. And if you go right here you're going to notice something this UL class. You're
holding lists. Each one of our entries is a list. After five entries you're going to have. You're
going to have i think it's 15 entries per page. This is the first portion that we need to start
considering to start pulling our data. This list holds everything containing our job description,
company, the city, possible salary, description of the job a hyperlink which you see right here. This
hyperlink if we use this takes us to the full job description which you could further parse
or do some text analysis or whatever you want to do. Post this on your website for example. Such as
the reason i was contacted to do this video. Let's start figuring out how we can start pulling these
data. Since these are all held within a list should i consider parsing this list by list? Should i
parse everything from the outer list? Make a call and pull everything my holds with inside of it?
For last year's video i was using the job scene beacon to extract these data. Today i'm doing it a
little differently. Some of the tags such as the H2 class that you see here. This holds your job
title. But you can also pull the job title in the span title right here. Depending on how you want
to parse we have to be considerate of something. Each one of these jobs held within a list also
contains a table. So this t-body is a table holding all of our information within it. You have to be
careful how you start pulling these data. Because you're always working from outer to inner. Remember
outer to inner for parsing and for scraping. Now let's go into our code real quick do some looking
around. If you want to follow along feel free to import these libraries here that you see. Otherwise
the code's gonna be on my Github which will be in the link in description below. The client had
specifications relevant to their system which are held within this header that they used. So if you
have a specific system requirement or something that you're working with. You can use this. That's
what's held within here. He decided that he was interested in the skill which is basically the job,
the location and then going through a specified number of pages. Iterating within this number of
pages. We can go through many pages. If you noticed here this Indeed is different
than this link here this link here. When i tried to parse i was having issues. He provided this
one which made it much easier for me to deal with. The skill, the location i decided to sort
by date for relevancy and then start pages. Every page after the first will have this 10. The first
page does not have this. Let's look at that so you can pay attention. We have a data analyst we're
looking for in Chennai Tamil Nadu. I'm sorting by the date and then you have this right here. Let's
go into the second page. And let me explain what i was talking about. Here's the second page and it will
have that start that we're looking. Scroll down we're going to third page and it'll have 20 and so
on. The next thing we're doing is we're taking our url and we're going to use the request library and
we're going to grab that url with these specific headers. And we're going to return this response
with text here. I'm using lxml because it's coded in C which would be faster versus the html which
would have been utilizing python specific code instead. Let's start from our beginning from our
outer body. Where we're going to basically use soup which is this variable to grab our website and we
want to find the div with the attributes of the id provider job cards that i showed before which is
the outermost portion of where we're going to pull these data okay. So we scroll back up and that's
right here. This is the outermost for each one of our entries. From there we need to work inside
with our UL list. I decided to iterate through this. Let's see what happens real quick okay. Let's
do two things. First let's see what happens when i decide to print the outermost point and see what
happens. Oh i gotta close that. So we're looking for a data analyst. We're going to take the city of
Chennai and then let's just do two pages. Here we go. Here's everything raw. We started from the
outermost we have all of our jobs held within. If we scroll down a little bit. We'll say okay
here's our job beacon. That's basically where we're starting for each entry. And we could say all right
so i see lid i got a title hidden in here. Here's a data analyst. And we have our HREF which is our tag
that we'll need to grab later. Let's get into all the other information. i'm iterating through the UL
tags. Let's see what this even looks like from the get-go. Let's go through this again and here's all
of our our lists holding our data. We should be able to see that here's our job title that we're going
to parse out next from this H2 class. And it is for full details data analyst for some company.
Let's see there's the title data analyst it's for Honeywell. Let's scroll down and look around
and this right here is the basic job description. This is the information that you're going to see
in this little text box right here. It's like the abbreviated version of what the job description
actually is. Now let's do something else. I chose not to go this route. And i'm going to show you
what's going on. So here this is relevant for each one of the job postings. This is just the basic
job description for each one of the jobs and you say well okay this is interesting. It's giving us
something for each one of the jobs but it's only giving us a brief description. What happened to the
rest of the data? It seems like i'm kind of not going in the correct direction and i'm actually
in a position where i might be complicating this for myself. So instead let's just go directly into
this and find our data. So we're going to find our job title like i showed before. Which would be held
within this list. We scroll down we see okay this is basically where we're starting from. Go down a
little bit and here is where i'm finding our job title which is highlighted on the left in blue.
All right let's print this out and see what's going on. This looks fine. It looks like each
one is delineated here. Let's see where the fifth one is. Because the sixth one should have
a blank. Yeah there you are none right there. So that's the division part that i was talking
about before. Where you'd have basically an empty list. That's perfect we can use this right here
to basically pull our job title, the name of our job. Basically we have a pricing analyst, we got
a senior analyst, we got a business analyst what else data visualization engineer, business analyst
analyst for learning specialist something like this. So we're getting closer. And if you notice
here we have this span tag which also has a position right in here. I didn't go that route for
specific reason. But we don't have to go into that now i created basically like an exception or like
a condition where if the job title is none. Because you have those empty lists after every five
entries it's going to be just a none or empty. We just basically can ignore this and we could take
that data and move in to finding our text. Which is in an A tag. Now let's print this off and see
what we're actually getting. We could store that because we could look at it again. We can look at
both actually and see what it looks like just put one on top of each other. it's just the one page.
We have a data analyst which came from this information here which was stored right there.
Because we pulled it out of an A-tag. This is basically just a raw portion here that has our job
title that we're printing out. and then i chose to pull the data analyst straight out of there. The
next position is this business analyst for this. Because they obviously did some weird formatting
for whoever wrote it. We got a risk analyst in compliance, we got a senior analyst we got
a data analyst. So we got that taken care of which is great. The next thing that we can adjust is the
company name. The company name is going to be held within the span class company name. and it
has the same type of situation after every five entries and then we have to go further inside.
Let's try to find this. So scroll down and it should be held within here somewhere .
So span title. let's find this company name that we're looking for we can start scrolling in here
and start digging around. Click on that and now we have our company name that we can extract
right here. And inside of it you see another A-tag. We have another one of these href's. But
this is a little different. So we said okay cool let's pull this company name it's bny
melanie within this a -tag. We were able to go inside this span class get the company name,
convert it to text, and just grab our companies. Now we have those links that i was talking about. Those
href links. Let's print this off real quick because this is going to be kind of interesting.
I want to kind of show what this looks like before you actually print this portion right. Let's
print both one on top of each other so you can get an idea of what is really going on. Let's just do
one page for simplicity's sake. Within that whole tag we're trying to find the href. Which is our
hyperlink. Which is this thing right here. Well it's actually this this entire thing right here. You see
like i think this is for mobile data i think. Like these links.That's what i'm thinking but i'm not
entirely sure. Because i didn't pull those. Now we go inside that a-tag and then we find the job
class title and we pull the href. Here's the job class title that we're going inside
and this is what we're snagging outside of it or inside of it. Next thing not every job has a
salary you barely find with these jobs at least for the pages i was looking at snippets that had
the job, the position salary. We have to account for that. When i was looking through this specific
class which had attribute snippet for our salaries we can see that when we look out of this you don't
really see anything. You see like one salary here and then the rest don't have anything. We have
to kind of figure all right where am i going to find this salary? If i'm interested in this in the
first place. But i chose to do it on purpose. So you could have one more little piece of information
that would help you out. This one has a salary and let's scroll down and start looking. There we
go a little bit more. There we are. So inside this heading we're going to start digging down deeper
and here's that div class attribute snippet that i pulled in the code. And inside of it we see oh
there we are. This is what we need to try to pull we need to snag this little bit of information.
So how are we going to do this right? Because it's right here. We want to pull that text with inside
of this so you had to go and just call the text and it'll snag it and then you basically have
to do an else statement. And say okay if we don't see that just say no salary. And then store those
in the salary. Make sure you see these are both the same so it keeps storing sequentially.
I decided to pull the dates because i wanted to give people the opportunity to practice with
date and time formatting as well. The date for the postings is in the span and we're looking for the
attributes with the class and the date. So let's just pull this one let's start scrolling down
this and figure out where we can see the lowermost portion. We're going to start about right here.
Start digging through say okay where is it at in here we start playing and there you go. There's the
date right there. Span class date. We can just pull the text out of this directly and that's what
we're going to use. Let's put everything together and append it into a list. I want to create a
default dictionary which allows me to take each key and store the values as a list. Let's pull
everything together and let's pull two pages this time and now we're starting to pull all of our
data. Okay, so what did i leave printed? I left these two printed. That's fine. Let's look at this real
quick and see what it looks like. We see that we have Honeywell needs a data analyst with this link , .
no salary and posted. Well it's just posted. Then we see all right Propel Technology. This is what they
need that's perfect. We have each entry stored as a list. Let's move on from that and get our
data situated. I did this real quick i just cut and pasted for simplicity's sake. But i took the
first entry from our list and i zipped it together with the list of what i want to be for each key to
give you an example of how we're going to create our dictionary with a default dictionary a list.
So what i did was iterate through the list from Indeed that we're creating and making
a list, zipping it together. Which is basically like what you're seeing in this prior step up
here. And iterating through that and creating our default dictionary list. Where this is our first
entry here. Which is our key and the second entry which is our value and that's how we're storing
our default dictionary list. Which says all right i have my company and here's all of our values.
This is good formatting that we can use later or for storing or we can create a data frame. If
we want to work with it now. And i want to show you something. Let's actually wait on this. Here's
the data frame that we're taking and instead of using the default dictionary. I just took directly
the indeed posts that were stored. And the columns. Column names which is the list that was created
prior. And you're going to see like the same type of jobs from different companies. Because some of
them are recruiters or things like this you're going to see overlaps. Because maybe they posted
the same job multiple times or something like this. Now we have this posted today. But this create a
actual date time formatting to take care of this. What are you going to need? These should
have been imported at the top but this is just for illustrative and reinforcing what you're
actually doing and what you need. I'm taking the data frame that we just created and i printed off
right here. We now need to iterate through the column of the job posted date. Which is
right here. I am using regular expressions to find all digits in each string for each row
of data. And i'm doing a string join. Where i'm finding all of these and i want to convert this
where i'm taking today's date and subtracting it from the time delta from the date that we find
and convert it into this string formatting of time. Now after i append it i have to also do an
else statement. Because you have two situations where you have posted today, just posted. So this
is taking care of anything else where we don't see like posted one day, ago two days, ago etc like this
and putting it all together. Here's everything that was posted as of today June 8th. This is
American formatting and be switched if you're in another country you know. For the two pages
that we got these are the job postings today and yesterday. This is what was posted. That's the little
bit of video that i wanted to show today. Pretty simple just to give you a heads up on different
things to think about when you're parsing Indeed. And if you want to compare and contrast look at
the other two videos that i did prior. Other items that i parsed and you can see the discrepancies
of what was changed and Indeed over the last two years. I'm sure that there's other ways you
can parse these data this is just one example. But thank you for watching please like, share
and subscribe. And feel free to leave a comment or connect with me on the community page. Thanks again
for the new member to my channel but i said at the beginning of the video see in the next one bye