What's going on? Everybody welcome to this second beautifulsoup tutorial in our little [beautifulsoup] [mini-series] in this tutorial, we're going to be doing is
Slightly more I suppose getting something really specific from my page where you can sort of think of it like navigation
But it's not totally [like] navigation like some other libraries have so anyway let's get into [it]
So I'm going to go ahead and just delete everything up to the soup
so
So first let's say let's say our objective was to get all the urls
but really we just wanted like a specific set of urls, so
You could say [right] for Url in
soup dot find underscore all a
And then you print them out and stuff okay, so we could do that route, but instead
What if we like maybe we rather than getting like the body?
Urls, maybe this is something we want to actually parse like the just that website we want to start navigating around that website so rather
Than taking the urls that are in the body text of that website
Maybe we want to do the NAv bar okay, so instead
We could say something like this right remember we could do soup dot p and that
Gives us like a paragraph tag we can do soup Nav and that will be basically the first Nav that it comes across so Nav
Can equal soup not now so now let's just to see what we've got. Let's print the NAV
and
sure enough, it's basically all of
the Nav here cool, so we'll close that and then now what we're going to do is [we'll] we want to find the urls, so
Actually the process to do that
I shouldn't have deleted that other thing
but anyway [for] a url in Nav find underscore all we will find all the a tags and
Then we can say print url dot get and we want to get a Chara
So this should give us all the links that it finds in the Nav bar
So here are all the links now just for the record. It's finding basically double the links because
Let me go to the website
We basically have two Nav bars on this website right this is one of the NAv bars
So it's like the the mobile version or small screen version right that's on the side here, but if you're not
How do I know how to make this go there we go?
Now I control my own website anyway if we make it bigger now the Nav bars on top
But that other Nav bars like there the codes there for it, so that's why it's actually being found so anyways just close that
so now we have all the links from the
The Nav bar now again also in many cases. You could do something like this
You could say body equals soup body
But in our case that's for paragraph and [souped-up] find oops actually body
Or let's say for paragraph in yet body dot find
all
p
We can print paragraph text, but we're going to find that this has
Actually that worked out better than I thought I was going to work out
Half isn't expecting that so actually that ended up working out, but we have two to two body tags
And I wasn't expecting [that] it was actually going [to] find that so on a lot of websites
They are like their content will be within a bot actually you know what I think I removed the body tag
Now that I'm thinking about it
Yeah, remove the second body tag before I went into here. That's that's what happen anyway
[so] so that's one option that you have so we were able to have to just get just the text
From the body and in most scraping cases like if you're just scraping a website
And that's you're just looking for some basic text data
This should be more than enough to just get that body data and so like if you're scraping for content
That's all you really need to do
The other thing you could do is like for [example] the reason why I got [rid] of the body tag is I added
this div and then
Class equals body well and sometimes you'll see something like this because there might be multiple body tags or there might be a really specific
kind of Div section that you're hoping to parse and
So let's show kind of an example there. So what what you might? Think to do is like, okay? Well, we could say
for DIV in soup dot find all
Div
we could try to print the dIv text and then I'm going to
See soup yeah, so we'll just do [that]
And then we'll print that
Right you're getting all [of] the dIV tag text. So that's kind of a pain
So one thing [that] beautiful soup allows us to do is actually have a div
But we can also specify a class you just class underscore since classes a class
so class underscore we can say
body there and
then you could say
for DIV
I'm going to run that really quick
Right, so that's good enough that gives us exactly what we were kind of hoping for so in this case
Remember this one is for a paragraph
Embodied find all paragraphs so that other [one's] not going to find the stuff. That's not within paragraph tags
Where is this? This is finding basically all of the text that's between div tags
All right, so that's like a little bit more about navigation. Just remember that you can you can basically specify a new
Beautifulsoup object right, so this is a the whole beautiful soup
Then we're saying here just that first Navbar and then we're able to just work between those navbar tags so again
It's problem. Maybe if you're not too familiar with working with
HtML. Right that's basically when we did a Nav equals soup Nav. We're basically dealing with
Basically here. Where are you Nav two here right just that chunk so we've sliced out that chunk of information
Okay, so that's going to conclude the second beautiful soup tutorial. We've got one more tutorial. I'm going to be talking about
Tables since that's probably one of the basically the three most common things I found myself needing to scrape is first the con
then urls then
Tables, so we're going to talk about tables in the next tutorial so stay tuned for that if you have questions comments concerns whatever on
This tutorial feel free to leave them below
Remember text-based tutorials on python programming net as well as the sample code [that] we wrote and I'll see you in the next tutorial