So now we're ready to finish the Web Crawler. And remember
what we want the Web Crawler to do. So, we have
a seed page, and we're assuming we know some seed page
to start with. And the seed page has some links on it.
We want to be able to find those links, and we
know how to do that now. We're going to get them to
list, and then we're going to follow those links. So we'll follow
the links. To new pages, and those new pages might also have
links, and we want to follow those links. So
in order to do this, we need to think about
two things. We need to keep track of all
the pages that we're waiting to crawl, and we'll introduce
a variable, tocrawl, to do that. And what tocrawl
will be is a list of pages left to crawl.
So, initially it'll just be the seed page. Once
we collect the links on the seed page well it
will include those links as well. Once we finish crawling a
page, we don't want to keep it in tocrawl anymore. And
as we find new pages to crawl, they'll be added to
the tocrawl list. The other variable we want is to keep track
of all the pages that we've crawled. At the end of
our crawl, this is the result. We want to know all the pages
that we found. That will be stored in the variable we'll
call crawled. So let's walk through an example of how this should
work on the sample site.
So I'll make the seed page, www.udacity.com/cs101x/index.html, that's
this page here. That means when we start
to crawl, we want tocrawl to be this
index page. And I'm going to stop writing out the
full URLs, just writing out the final part.
because all the pages we crawl will be
on our test site. So tocrawl will be
the list containing just one element. The index.html page.
We haven't crawled anything yet, we're just getting started, so
crawled will start out as the empty list. The next
thing we're going to do is crawl this page, so
we'll get all the links on this page. That means
we've crawled the index page, so that will now be
added to crawled. But when we looked at the links
on the index page, we found three new links on
that page. We found a link here, which goes to crawling.html.
We found the link here, which goes to walking.html.
And we found the link here that goes to flying.html.
So the new value of tocrawl, after crawling this page,
will have those three links in it. The next thing
we want to do is take one of those links
and crawl it. The order actually matters a lot in
terms of getting a good crawl. Let's assume for now
that we'll do just do the last one first. So,
we'll do the link fly. That links to
the flying page. Here is the page there. So,
we are going to crawl the page flying.html. This
page doesn't have any links. If you're not sure
why squeamish ossifrage is the magic words, I
would encourage you to DuckDuckGo or Google that. And
now we've finished crawling flying, so that's going to
be added to the crawled list, which already had
index.html, we don't lose that. We're going to add the
new one, which is flying, to that list. And
we finished crawling it so we don't want to crawl
it again. Let's remove it from the tocrawl list. Now,
after we're crawled flying, we have two more links
left in our tocrawl list. We have two links
that we've crawled. So let's try another link. Let's
suppose we follow the crawling.html link. And we follow crawling.
We get to this page. So, to follow crawling, we're
going to follow the same algorithm we did with flying. Alright, so
that removes this link from the tocrawl list, adds it
to the crawl list. So, we're done crawling, crawling. And now
we want to add to our tocrawl lists all the
links that we find on this page. Well, we found the
link, kicking, which goes to the page, kicking.html So we're going to
add that to our list of pages to crawl, and now
we keep going. And we're going to keep going.
We'll follow kicking. We find that kicking does not
have any links. So that would add kicking
to the crawled list and remove it from the
to crawl lists. And we're going to keep going
until we have no more pages to crawl. So
let me describe that process a little more formally
and then I'll ask you a question about it.