Hi.
My name is Emily.
And today I'm going to be continuing off
from my previous video that introduced you
how to create the basic skeleton of a web scraper.
So if you haven't seen that yet,
I recommend watching that first.
Otherwise, just to remind you of the problem,
we have this 'util_scraper' program we're trying to create.
And we have created the skeleton for it.
So we've gotten the command line arguments into a loop,
and we've gotten our file downloaded
into our file handle.
And right now our main loop is just
reading through those lines, one by one.
But, as we can see from what we've
downloaded in the previous video,
the HTML files that come are quite complicated.
And there's not really--
you can't really just point to the part of the screen
that you want the information, like you
can with the timetable information on the website.
So we need some sort of way of identifying patterns,
to spot which lines we want to extract
and how we actually get those percent-full statistics.
So what I like to do, whenever I'm faced with
this problem, is, again, just look back
at the rendered version,
so I can spot which parts of the website
I'm actually aiming to look for.
So, if we search for "COMP2041" here,
we'll see the first hit is surrounded
by a bunch of other course codes.
That doesn't give us the percent-full statistics.
So if we see it on the HTML code,
it's probably not what we want.
The next one is down here, in the table.
And we can see-- oh, look, that is
a percent-full statistic I want.
So we know that, if we get to the second part,
where we get a bunch of lecture information and 100%s,
that's pretty much where we want to
start looking very carefully.
So let's look back at the HTML file.
If we go to the top, and I search for 'COMP2041',
the first hit is a bunch of files, over here,
with a bunch of COMP course codes in a table.
And, if you're worried about your HTML,
if it's a little bit rough right now, don't worry.
Just look for "W3 Schools HTML"
or some other similar web resource.
You don't need to understand HTML fully to do this exercise.
You just need to kind of understand what the tags are.
And the most important one is really just this 'TABLE' tag.
So if you see 'TR', 'TABLE', or 'TH',
that just means we're organising information.
So a table, there might be rows, which are 'TR',
and 'TD', which are "Table Data" items.
So, if we look at our COMP thing, this is a table row.
Each item, like '1UGA', '2556', those are table data items.
And that is reflected over here.
So, in the first part, we have a bunch of table data items,
with a bunch of course codes.
And, looking back at the website,
that's probably this area that we have
identified as not what we want.
The next thing that we search for is in this section.
And it looks really complicated, but let's just take a breath
and look through the code.
OK.
So we see 'COMP2041' as a name.
Afterwards, we see a bunch of 'LEC', '1UGA', percent sign.
So that's a pretty good sign.
That's where we want to be.
But, as we know, we can't just search for 'COMP2041',
because you might have multiple things that come up.
So when we searched up 'COMP2041' the second time,
this is what we got.
But the first time, up top, we got this line.
So we need some sort of way of differentiating the two.
And, looking at these two lines right now,
the thing that jumps up most at me is that we have
a 'class="cucourse"' in the second line.
So classes is something that HTML and CSS can use
to identify different sections.
And we know the first section is quite simple,
there's no class over there, but I'm going to make
the hypothesis that every time I have that table row header
I have a 'class="cucourse"', and I have my course code
that I want in bold.
So let's just double-check that assumption
by searching through our HTML file.
So, if we search for 'class="cucourse"'
and we're looking for a name in bold, I get 'COMP1400'.
That's good.
But I know I can't just use 'class="cucourse"' by itself,
because the next line also gets highlighted but that's not
a new section.
So keep on going.
Over here, I get 'COMP1511'.
That's good.
'COMP1521'.
So it seems like that is the pattern I want in my regex.
So the first thing I want to do is identify--
I want to just skip past all of the lines
until I get to 'COMP2041'.
Because this, that's the section
that actually makes sense to us.
We want to get to 'COMP2041' and then get to this percent sign.
So everything before that, and everything after,
I don't care about.
So in our 'while' loop, we want to identify
where that is. So, if our '$line' has 'class="cucourse"'--
which is what we identified as being part of what we want,
but we also need to have the course code in 'B',
and the course code is in our variable course.
So, if it has those two things, that
means we are in the right section.
So we want to set some sort of flag variable for that.
Let's call it '$found'.
So outside the loop, let's say '$found' equals 0.
When we're inside, set '$found' equal to 1.
And, if that's the case, that means we're in the right section.
And we know that, from there, we need to look for all the 100%
or whatever percent signs that are on the same line as a 'LEC'.
But looking at this, you have to be careful
because some lectures might be closed,
and that might give us false hopes that we might
be able to get into COMP2041 when it's actually
a closed lecture stream and there's actually no space.
And if we scroll down a little bit,
you also see that lectures sometimes come after lab.
So we can't just say it's the first few rows of table data,
because lectures might be at the end
and it might actually be scattered everywhere.
So that's not standardised, so we can't make that assumption.
The only thing we can make is that it's
going to have a 'LEC' on that line, it won't have a "Closed,"
and it will have some sort of percent sign.
So let's confirm that with our code.
So we go back to our 'COMP2041' section.
So 'COMP2041'.
Get to the second one.
We see that that's our line that we've used to flag
we are in the right section.
Afterwards, we want to keep on looking,
and the lines that have the information we want
look like this.
They have a 'LEC'.
They don't have any "Closed"--
any "Closed" words, on that line.
And it has a percent sign, to say what
percent capacity that we're at.
And, if we look at the next one, in this case there's a 'LEC',
but there is a "Closed."
So we want to make sure our regex does not
include those cases.
And those percents signs, we also don't care about.
So looking at this, I have my idea
of what I need in my regex.
So let's write it.
So in our loop, if we are past our '$found' section--
so we've passed that critical line that tells us
we're in the right place--
we need to see if the line has 'LEC'--
all in capitals, spelt just like that.
And we need to make sure it doesn't have "Closed."
And, if it doesn't have "Closed," the last thing
we need is, does it actually have that percent
sign that we're looking for?
So we need to see if line has something, some numbers--
so, one or more digits--
attached to a percent sign.
So that's what we're looking for.
That means we've found the percent-full line.
And, if that's the case, we can extract that percent full.
And we can do that again with regex.
We want that 0 to 9 plus percent,
but we want to capture those numbers without the percent signs.
So let's put our parentheses like that.
That's our percent full.
And, looking back at our prompt,
we want to output the course code, "lecture is",
how many percent, "full".
So to format that 'print' statement, we need 'print',
our course code, "lecture is," '$perc_full', "% full."
That's what we want to be printing out.
That seems to be all right, but let's test it out.
So we use our 'util_scraper'.
Let's just use one course code, for now, to simplify things.
Oh, no!
We have a bunch of weird output over here.
So right now we've always got 1% full,
but we probably want 100% full.
What's going on here?
So let's try adding some extra parentheses
over here, we want our line to match when we have
one or more digits percent.
So let's try that again, see if we get
something comprehensible.
So we needed those parentheses to wrap it.
However, looking at this, we apparently have
a lot of COMP2041 lectures, when we can see from our website
that there are only two full ones.
So we only ever want to match two, which are both 100%.
What's going on here?
So if we look at our HTML document,
we can see, OK, let's look for LEC.
Those are all lecture streams I want to match.
But if I keep on going, I'll actually
go into some other lecture streams.
But this lecture stream is actually
for a different course.
So we have specified when we are in the critical section
that we want to be looking for lectures streams,
but we didn't specify when we should stop looking.
So that's an extra 'if' statement we have to add.
And let's just see when should we actually stop looking.
So when we want to stop looking, that means we want to be...
past this critical line, over here.
And then we keep on scrolling.
And you want to get to the next critical line,
for the next course code.
So if we keep on scrolling, we want to get to--
so this is microprocessors.
And then, again, we have a 'class="cucourse"'
with a different course code, 'COMP2121'.
So we know that we're in a different section when
we're identifying 'class="cucourse"',
because we said that's how we identify the big header rows.
But it has another course code, which is not 'COMP2041'.
So back in our regex, we can add another case.
So in our case, we want to say, OK,
if we are in our critical section,
but we see the same 'class="cucourse"'
and there is a course code in bold--
so it's some random course code.
It's not the one we're looking for.
So four capital letters, four digits.
If we see that, then that means we are in a different section.
And, if we're in a different section,
we have already gotten all the information we need.
So we could just simply break out
of this 'while' loop, using the 'last' command.
So hopefully that fixes our problem.
Let's give it a try.
'COMP2041'.
So we have an uninitialized pattern in match on line 24.
So let's look at line 24.
If '$found' is 1, if '$line' is equal to 'class="cucourse"'--
And we didn't specify the line, here.
So, '$line'.
And then we have that pattern that we're looking for.
Let's try it again.
So there.
We only have two lines, which is what we expect.
So that's so far, so good.
Let's compare it with the output we've
said is supposed to be correct.
So if we have util_sraper, and we put "PSYC1011," "COMP1511,"
"COMP2041," "ECON1102," we get the output that we expect.
So that's pretty good.
Our program is working well.
So the big takeaways is, first look at the timetable,
the ClassUtil website, to figure out what the pattern is
in a format you're familiar with.
Then look at the HTML file and see
how do you differentiate lines of code that look similar
but you'd only want one of them.
So, looking for patterns and seeing what
differentiates them.
Then it's simply a translation exercise into Perl.
And then make sure you test, test, test to get rid of bugs.
So that's your basic web scraper.
I hope you've learned a lot.
However, there is room for improvement.
So, if you want to see how to extend this,
using options for printing out different statistics,
watch my next video.
Otherwise, I hope you have learned
a lot from these videos.