what's going on everybody and welcome to
a new tutorial slash some coverage of a
new package called requests - HTML it's
basically just a way for you to really
quickly easily parse some HTML so it's
written by the same person who wrote the
request library so my expectations are
high but this is a very new package it's
been on github for about a month maybe a
little over so I would expect some rough
edges but let's check it out
so first of all to install it let's
scroll on down to the bottom to install
it you just pip install requests - HTML
you will need Python 3.6 or I'm assuming
later but right now the the latest
release of official python is 3-6 so
let's make sure you install it and then
let's go ahead and check it out so to
check it out I'm going to be using the
following web page at least to start
it's just Python program it's slash
parse me make up our space and basically
it's got some tags so list a table it's
got imagery it's also inside of div tags
just so you can kind of poke around that
it's got some JavaScript tests so if the
JavaScript loads it says look at you
shine in and then if it before it loads
its default is like why you bad though I
think then we've got some pre tag
information in here with the Zenna
python and we've got a link down here
and then we've got some goofy looking
characters here that it could throw you
off so with that this is what we're
gonna start off by parsing so coming
back over here it looks pretty simple
I'm just gonna like copy and paste this
and just remove the interactive
interpreter - things so it should be
that simple so you can request get I'm
gonna pass in my own URL you can feel
free to use something else if you want
I'm just going to be exemplifying some
of the things I use this page because
for this it's a one page I can control
what happens to it it seems like every
time I do a tutorial on literally
anybody else's pages or with anybody
else's API they all get deprecated they
all change so this one should be a rock
now so that starts your session we get
some stuff and then we can start
actually interacting with it so again I
don't really know everything with this
package but generally when I check out a
package
mean they might I'm not sure if he has
like a documentation page it doesn't
look like he does let's see oh maybe he
does let's see what this looks like
it's pretty much the same thing for HTML
this is interesting
pagination that's uh that's more magical
than I would have expected that's
incredible
if that really is that simple to do
pagination wow that's not written up
here okay so anyway that's interesting
to say the least so then you request the
oh mine and then you can just iterate
over it oh that might work really well
with Reddit I'm curious to know if that
works well and other websites we'll have
to find well it's a test that maybe at
the very end you can use it without
request so if you had downloaded maybe
some HTML documents you could then parse
them yeah this has a lot more
explanation Wow okay cool
I'm really that's that pagination so for
that's just icing on the cake man I
would think you'd have to find the the
link to it and then go to the link the
fact that the pagination I'm sorry I'm
that's just that's just interesting so
back to the simplified version of the
docs though all we need to do is once we
get we can just start referencing it I'm
pretty sure as long as it's not
JavaScript so yeah immediately we can
just say like for example let's just do
our dot well first of all let's print
Durer ours so let's just see all the
things that we could do with it let me
just pull this up so immediately on the
request we could check I guess the
encoding I get a parent encoding we can
close the request cookies and coding I'm
not sure with the difference between
encoding a parent encoding is check the
headers history we can reference the
HTML
we can check if it was a redirect that's
interesting so it just automatically
does redirect you but you can find out
if you've been redirected that's
interesting JSON I wonder if you would
just pull it I'm not sure if you said
dot JSON like if it is like if it's like
some sort of API or something in the
responses JSON if you could say JSON
it's a JSON object now I'm not sure I'm
guessing that's all the links that's if
it's paginate 'add apparently which is
fascinating also be cool if it wasn't
paginating I wonder if you could make
some sort of crawler that just like
automatically goes to any link and just
keep slinking around if anybody that's
trying to build just like a web crawler
that's kind of an annoying thing to have
to build okay cool so we've got text URL
and then the HTML so let's check out
let's just check out dot HTML so first
of all that's I guess we'll print out
the Derby HTML as well but let's just
see what so we've got text raw HTML
search so we could search the HTML I'm
assuming links find all of the links so
like for example it's print dot HTML dot
links so what that looks like okay so it
looks like it gives us a set now I guess
that makes sense to give it a set just
in case there's multiple of the same I'm
guessing that's a set I don't know we
can type just looks like a set yeah so
it gives us a set of the links what else
could we print out our links we've got
our dot HTML
wasn't that adder of our to HTML ooh I
think it's all formatted and beautiful
for us so what's the difference between
HTML and raw HTML oh yeah ok that gives
us like the new lines in like all the
tabs and spacing and stuff like that
okay
okay I'm not sure exactly why you would
need to do that but okay so that's
interesting the other things let's see
let's check out
find you can find it looks like ah the
IDS alright I'm not sure let me see if
if we have IDs here I know at least we
have probably the ID in the yeah yes no
js' I wonder if we can find that one I'm
just gonna copy paste are not find yes
no js' print about even though that's
kind of bad words ah darn it
I see uh probably text I'm just guessing
nice okay so that's how we could find it
I don't like what it's saying to me but
that's how we can find it which then
brings me to the next curiosity I have
which is with the JavaScript rendering
so what's happening here and first of
all let's just say I don't know Jay s
test there's copy that paste that here
pull this up save okay so at least the
whole point of this was to test if we
could actually load JavaScript when we
parsed things since that was part of an
older tutorial but now with request HTML
apparently it's super simple so let us
check that out so if we scroll on to the
documents it looks like basically all
you got to do is just use dot render so
so let's go ahead base it right after
the request so let's just paste that in
there so with JavaScript the immediate
return will be the base javascript
including like the actual JavaScript
code and in order to render JavaScript
you actually have a browser that runs it
like that's how JavaScript works like
you request the server and the server
just responds in just text obviously and
then your browser is gonna see you okay
here's some script we want to run this
script so what you have to do in order
to read elements inside of tags that are
updated by JavaScript is you need to
render it and you also have to kind of
wait it a moment you have to wait until
it's done rendering so that's kind of a
tedious thing to build yourself but
apparently it's in in here and ready to
go so let's go ahead and run it
and let's see if it updates it so
on the first run it looks like you have
to install chromium so I get scan it
download it download may take a few
minutes I hope it doesn't I don't really
want to wait that long
maybe I'll pause it while we go oh I
think it's done okay it's done okay and
it's just telling you where it extracted
to mmm and we had an error unable to
remove temporary user data interesting
it did work though that's the that's
this is the text right here that we were
looking for so like if we just kind of
make some space here since the errors in
our way it's probably got permissions
there honestly
anyway yeah that's what we were looking
for and that's indeed and you know if we
come over here so it definitely works
but for some reason we're getting an
error when we attempt to remove the user
data so let's open up this file the
launcher so at least in sublime it's
really nice you can just double click on
the thing that airs and we could check
it out okay so in this internal method
we're trying to clean up some data and
they're using Shu till remove tree so
it's trying to remove a directory
basically that's how you're going to
remove directory with contents huh so we
already we saw where it went it goes
into users H and then that let's see let
me see if I can find it
yeah so it goes in you know here it is
and then if we go into there we can see
that the contents but for some reason
we're unable to delete them I'm gonna
guess it's a permissions error one way
we can confirm that is to not ignore
errors we'll set that to false
just in case anybody's like oh why would
you ever ignore the errors I forget that
I think if you don't ignore the error I
think one of the errors is that the file
directory has contents and that'll stop
it from working if you ignore errors it
does work but we also don't get to see
the error so let's go ahead and see cuz
things aren't working but once you once
it is working once SH util does work for
you you actually kind of wanna yeah
access is denied you actually do want to
put ignore errors to true so let me just
set this to true again so we're seeing
here access to tonight just we don't
have the permissions I'm kind of curious
I kind of want to open up an
administrator to see if I can break that
let me pause for a moment let me open it
up and head to where we're working and
see if we can use at least command
prompt as administrator and see if this
works okay
let's give it a shot we're administrator
control panel Wow let's still can't
remove it we did get the info here
though
hmmm but it's gonna break any script
that's running too though that's really
quite the bummer at least right now the
show must go on so I'm gonna continue at
least a short-term fix that I can think
of I'm yeah why wouldn't that work I'm
pretty sure if you used your own like
Linux and you sudo though that this
wouldn't cause any problems for you it's
maybe it's the location I'm not really
sure I really can't decide why that
won't delete if somebody has a better
idea go for it otherwise temporarily one
option we have is to just not raise the
IOA don't raise the io air instead we
could just print we can just print
unable to remove like we don't actually
have to we don't need to delete that
file we just would have liked to to keep
things clean alright let's try again
cool okay so now it'll still at least
print out to us that something went
wrong but we would be able to continue
so like if like for example actually I'm
not even sure moving already moving
along was that a that's kind of weird
cuz the other print actually I think was
working so we are moving along maybe at
maybe it wasn't I thought there was
space being made that doesn't make any
sense to me though if it would raise
that error why it would continue working
wow it actually does okay magic I guess
we could leave the IO air there to be
honest because it does can we were able
to continue moving along I don't know if
that's bothering you though you can get
rid of it let's see what else we could
do so I'm curious to parse that table
also let's try to parse let's purse
Finance dot yahoo.com that sounds like a
fun one
I'm pretty sure they block automated
requests now okay so what we can what we
can do is like if we go to amazon.com
one of the things I used to do is parse
from here and if we go to like pry
statistics right there's all these
statistics and for a while they were
just I'm not even sure how they did it
to be honest but it must have nodded
it just wasn't with JavaScript but now
it is with JavaScript and these values
are not like static they're updated via
JavaScript so and now I'm kind of
curious I wonder how they did do it like
I guess you could like if you use like
Jinja formatting let's say you wouldn't
necessarily use javascript to update
those values so maybe that's it anyway
let's head to this page booth and now
what I'm curious about is like can we
find I wonder first dot true if we don't
do that if it'll find us a list like so
for example
what I'm gonna be looking for is like
let me just do it I'm just gonna go into
the source code and I'm gonna search for
forward p/e so it looks like we could
look for any table data for example like
so like what if we wanted to just find
all table data so probably we have to
use a tag as my guess let's see if they
have any tags here so about text links
okay find a so this would be any so I
wonder if we could just do find TD like
fine table data about HTML about
attributes about I think you can just
use dot find in any bit of test text so
let's try this let's say or dot render
let's do print or dot finds let's find
all the table data let me just see what
we get because I think we'd actually
have to be table data text probably I'm
not really sure we're still gonna see a
stupid message to HTML response object
so probably dot HTML maybe it needs to
be hard that's weird
all right yeah so it needs to be our dot
HTML finds probably it's hard because
they don't have just like searching
purely HTML but I'm pretty sure you need
to do the HTML yeah yeah cool wow that's
super neat so you can get all the table
data and then in this case all it's
super annoying with their table data in
the Yahoo is each one is has a unique
I'm pretty sure or does have a unique
class they are actually all the same yet
they're all the same class fine table
data and I wonder if we can search for
the class specifically to in in request
dot HTML dot find that would be
interesting to know if we can do that
I'm sure you can let's see yeah oh well
you have class at least there where's
the UH where's that link to the full
documentation oh right here cool let's
see if we can do a find first equals
true about class about time attributes
like I wonder if we can find specific
classes I'm sure there's a way I'm just
trying to figure out what the parameter
would be wait
oh so could we say TD dot let's just try
let's try ooh what about with the spaces
though I guess it yeah hmm I don't think
this is gonna work I don't have high
hopes for this one
I wonder I think it would find him if it
didn't have spaces but let's just see
what happens I kids know
and now I don't even know what the other
error was because we ran out of space
apparently oh maybe this is it there
just wasn't anything hmm
yeah it looks like the parentheses threw
it off bummer
not really sure I'm going to go ahead
and move along from that one but that's
interesting this is we're probably going
beyond what it intended us to use this
for let me see if any of these have a
class real quick like I bet I can do and
maybe that dot was for the ID I'm not
sure let's do let's go back to parse
mimic parse face just curious find I
want to find all the div tags give that
modal class no we really don't need to
render it anymore that way we won't have
to see that error anymore yeah cool
anyway let's do that so if you wanted to
find all the like of a specific class
you would use dot cool okay anyway I
think that's enough for now this is
probably getting pretty long that's just
me kind of poking around with the
library if you have any questions or
something like that feel free to share
if you've got some cool things that you
found about it you can share that too
I'm also pretty curious with the reddit
thing I take its word for it I'm trying
to think of something real quick that we
could search for the pagination I just
want to see that work on everything I
just understand how you could know that
it was pagination I just can't think of
something that that I could search real
quick that has pagination to it but I
wouldn't mind poking around with that
I'm gonna guess it works at least on
reddit but how would they know how do
they know about the pagination
intelligent pagination support always
improving so I'm guessing it doesn't
always work what if we did what is it
let's do sue hacker news that's got some
pagination to it so what if we go to y
news dot Y Combinator calm and then
let's see how they did it so they're
just saying our dot HTML next so what if
I say this our HTML dot next print
really has no attribute next let's check
reddit real quick let's see if that one
worked that's kind of weird
why wouldn't no that one doesn't work
either well then I'm not sure for our
HTML hmm let's see if this one works
like I'm just trying to do the example
that they've got posted now for HTML
let's just do this go away
whoops
what did I do wrong are equal session
docket are equal session that guy yeah I
read it and then this two lines don't
forget that there goes so it gets the
initial one and it definitely is getting
the pages and I guess each time it goes
to that page to get the next one which
is interesting so it definitely knows
that pagination let's try again on
Hacker News because the next thing the
next operator thing doesn't appear to be
working
Oh started to work they failed so we got
to page two and then it went to CNBC so
I think it just picked the wrong page so
you started here and then we go probably
it found page two which was smart and
then somehow I got stuck on CNBC not
sure why so okay the pageant okay it
sounded too good to be true and I think
it is but hopefully over time it will
improve and that'll be like that's
pretty cool anyway I think now is a good
time to stop
I mostly ran into the edges of this this
package but for simple HTML parsing and
stuff like that this clearly is super
useful you won't need to use
beautifulsoup and stuff like that but at
least we found some issues with the
javascript thing it just can't seem to
delete that your temporary directory so
eventually you'd probably have to go in
there and delete it just manually
otherwise i'm trying to think of what
else obviously like the iteration didn't
really work but still the rendering
actually worked and i'm not sure you
would hit that issue on linux so just
the fact that i'm on windows could be
responsible for that issue in them the
pagination i mean come on that's really
hard
i'm surprised that they're even trying
to do that but cool at least it works on
reddit too bad reddit has
P I'd probably rather use that than
paginate through them other than that I
think that's it you like the search is
super useful super quick super simple
yeah so pretty cool package if you need
to parse HTML I think this would be the
way that I would suggest you go well
yeah if you've got questions comments
concerns whatever feel free to leave
them below otherwise I will see you guys
in another video