In this question, you were asked to limit the depth the crawler searches to.
In order to understand this question and what we need to do,
I'm going to draw a graph of the links, so I'm going to start off with a seed page A,
and from A I have links to another page B and another page C.
Now B links to D, and C links to E.
Now, just to make things a little tricky, E is going to link to D.
Finally, D links to F.
What do we actually need to do?
Well, we need to keep track of what pages we have left to crawl, just like before.
We need to keep track of the pages we've already crawled,
because once we've crawled them we don't really want to have to crawl them again.
I'm going to introduce a new list where we keep track of the pages at the next depth.
I'm just going to call that next_depth.
We start off at tocrawl, and we have just the seed page.
Now, we haven't crawled anything yet, and we don't know what there is to come either.
We start off with just tocrawl with the seed page in it.
Now, from there we look at A, and we see that there are two pages that A links to.
We're at A, and the next depth--the next pages at depth 1--are B and C.
We're actually going to move A from tocrawl to crawled,
because we've actually looked at it now.
We've seen that the next pages are C and B.
Those are going to be the ones that we crawl next. I'm going to put those into tocrawl.
We'll look at those next.
If we look at B, B has a neighbor D, and then C has a neighbor E.
We do this in a couple of steps, but just to get the idea how the code is going to work.
I'm just going to go through it looking at what's at each depth.
B and C we've now crawled, so those two both go in to crawled.
Now the ones at the next depth are D and E, so those are the ones we want to look at.
We're going to move D and E over here,
and then we're going to look at the next pages of D and E.
D has neighbor F, and E has a neighbor D. Those are at depth 3.
Now, we might not want to add D in here, because actually we've already look at D,
but we'll see when we get to the actual code how we're going to handle that.
Now we take D and E and put them into crawled.
Then we've got F and D over here,
and we can carry on, depending on what depth we want to go to.
That's the idea behind how the code is going to work.
We're going to start off with a page.
We're going to crawl it and everything else at the same depth,
and then add those pages to next depth and switch the lists around,
which hopefully you'll be able to see more clearly in the code.
This is the supplied code for crawl_web, and we're going to make a few changes
to this to take account of the depth. So what do we do first?
Well, we're going to need another list next depth,
which will keep track of the next level of links.
When we start we have a depth of 0.
We're going to carry on through the while loop while there's something in tocrawl,
because if there's nothing to crawl, then we can't carry on.
Also while our depth is at most the maximum depth that's supplied as an input to the function.
Just as before, we're going to pop from tocrawl,
and we're not going to make any changes to this line either,
but instead of adding to tocrawl--because we don't want to muddle up
the pages we're searching through now and the pages at the next level--
we're going to actually add the new links to next depth instead.
Then just as before, we're going to add our page that we've crawled
to the crawled list so that we know we've already looked through it,
and we're not going to look through it again.
Now, the last bit of code that we need to add--
when we've finished going through all the lists at a given level,
just like I showed you in the example,
we're then going to change our tocrawl list and our depth list over.
So tocrawl is now empty.
Next depth contains all the new links, and I'm going to use a multiple assignment
where I swap tocrawl and next_depth.
Tocrawl becomes equal to next_depth and next_depth is going to be empty again,
so that we can start the while loop again with a clean list.
Then depth we're going to increase by 1.
Then we're going to return crawled.
Just to quickly recap--we carry on through the while loop as long as we've got some page
to crawl and our depth is less than or equal to max_depth.
We take page, and if we haven't crawled the page before,
we add all the links from the page to the next_depth.
Then we take our page, and we put it in the crawled list.
We keep doing this. If we haven't got any pages left to crawl, then we switch them around.
But if we have got pages left to crawl, we just carry on with the while loop
until tocrawl is empty.
Once tocrawl is empty, that means that that level is finished, that depth is finished,
and we want to go on to start the next depth, and that's it.