hello world this is cs50 on twitch my
name is Colton Ogden and today for the
first time we're joined by Emily Hong
hello welcome to Emily what are we
talking about today today we're gonna
talk a little bit about web scraping
nice what is web scraping yeah web
scraping is basically you have the whole
world wide web right and I know as a
normal computer user you can kind of
like go on a website and read
information and so kind of there's so
much information available on the web
and web scraping is a way to kind of
programmatically take that information
off of those websites and do whatever
you want so sort of parse it
programmatically as opposed to an end
user who might be going on Wikipedia or
Google interacting with things sort of
with the mouse reading you know
information so we can make a script to
maybe do some analytics and sort of yeah
so if you think of like maybe the non
tech way or like same concept of web
scoping maybe you're reading a website
say Wikipedia and you're like copying
down the words or information on paper
and then we decide to do something with
paper that obviously takes a really long
time fish in I was yeah and so with our
computer science knowledge we can do
some neat little tricks to speed up that
process are we gonna be using Wikipedia
at all by chance in hell yeah yeah fun
fact also Emily and I are from the same
exact well not the exact same place but
same county very in California which is
amazing yeah let's let's bring it over
to your computer here just so we can see
that and hopefully we did a little bit
of cropping in advance hopefully though
actually I didn't realize that might
screw up the slide for example we don't
see the way the image if we're gonna
stay yeah that's fine I just wanted to
keep everything off of that but there is
an image credit there and although I
just think that Emily originally had
created a image herself which is it's a
very beautiful representation of what
beautifulsoup yeah exactly
so what is beautiful soup actually
because I'm curious yeah so we talked a
little bit about generally what web
scraping is that's kind of like a
general computer science concept but
then there are different ways to like
carry out that idea and so today our
little
intro demonstration is gonna be doing
web scraping particularly with Python
okay and beautifulsoup is one particular
library that someone has so graciously
made and like allowed us to use and they
called it beautiful soup kind of because
they visualized all of the internet as
kind of this huge unorganized soup an
alphabet soup exactly like an alphabet
soup like on the screen and a beautiful
soup will help us make it beautiful
among others is this sort of like the
most popular I guess web scraping
library in Python
um I am actually not a hundred percent
sure I do know that it's one of the more
easy to use ones so maybe not as heavy
duty as some other ones that you can
find but very like friendly to
first-time users I'm very excited to to
dig in here a little bit it's making me
a little hungry too but we have a few
people in the chat let me see some folks
are asking what is the difference
between here and EDX here likely being a
reference to cs50 on twitch this is just
a twitch channel where we take in guests
and we have from scratch implementations
of projects we talk about concepts EDX
is more like a traditional lecture-based
set up a lecture based course platform
for example cs50 is on EDX but also of
course that I taught at Brian taught at
Jordan thought those are all on EDX so
it's a little bit more of a traditional
setup this is more a collaborative
teaching approach and you tido 988
asking was the topic of today web
scraping with beautiful soup hence the
hence the title slide there is beautiful
soup Jay s pup
what is it a JSP up is it a thing I'm
not entirely sure what you mean by that
X Polar dreams oh maybe it's like Oh Jay
soup Jay suit is like a JavaScript oh
it's a java HTML parser probably similar
imagine it's very simple I'm not sure
I'm not familiar with it but it looks
like in based on at least what I can
read here it says open source java HTML
parser with dom CSS and jquery like
methods for easy data extraction sounds
to me like it's probably yeah I have to
imagine it's probably a like a cross
language version of this idea that's
from yeah yeah probably
yeah good questions all around and then
Shen wen was saying hello Emily and
Colton awesome why don't we dive into a
little bit of alphabet beautiful soup
yeah it's all about yeah so we talked a
little bit about this but generic yeah
sorry I took we have a little bit of our
chat windows a little bit large I don't
wanna cut off your slides okay whoops
let me go ahead and shrink this down
just a little bit like that boom that
should be okay um okay generically web
scraping is the idea of using computer
software work code to extract
information from websites and that's any
website that you as a user could access
normally okay and so the idea with
beautiful soup and no more chat we're
gonna we're you're gonna hide that for a
little finger back apologies but with
beautiful soup is one the idea of making
maybe unorganized websites more
organized so if like I know when in
school like sometimes I'll look up like
a at more niche academic concept and
it'll view some like plain HTML page
which are some text like saying you
wanted to take that content and maybe
like spruce up the the look of it make
it have like fancier buns or whatever
and you could do that so that's the idea
of bringing unstructured data or maybe
like poorly structured data and making
it like more user friendly sure the
other idea is there's just a lot of
information on the website or on the
Internet and that's constantly updating
and so you can grab that information
through some kind of what's creeping
through that HTML page and put it in
some kind of database or spreadsheet
that you can then manipulate on your own
sure almost taking like other people's
database information putting into our
own database yeah yeah we'll talk a
little bit about some things to consider
before you scrape by web scrape well I
kind of I think everyone knows there's
so much information and content being
produced on the Internet these are just
something like I don't know little
statistics with if you can imagine all
the kind of stuff that you can access on
1.8 billion websites I'm actually kind
of surprised I almost expected there to
be like maybe 10 times more than that
yeah this might be a little outdated
there's let's see if this website is
still alive and also a shout out we have
a couple followers too
Tasha pen well cool game war 12 and a
meet Microsoft India thank you very much
for the follows oh wow okay so an actual
live web site that shows all this stuff
this is internet lifestyle really not
sure how beautiful is it but it looks
pretty professional random number
generation it seems it seems legit so
let's say that Wow one hundred and forty
six billion email sent just today so we
probably can't access that kind of
Internet information because it's not
publicly available but that is happening
on the Internet Wow
this is fascinating okay yeah so just
some things to consider the next time
you post something on Facebook I'm a
little surprised at 800 Instagram posts
per second - I almost feel like probably
before but again let's say at least um
so why are web screen there's again just
all kinds of stuff that happens on the
internet something that's particularly
interesting you could say like oh why
don't I just like download the HTML page
get the information then I never have to
deal with it again sure but there are
also a lot of websites that constantly
update right so an example that always
comes to mind is like flights prices
although there are a lot of like other
services that centralize that
information for you now again like maybe
a blog post maybe there's someone that
is posting regularly and you want to
take their information - I don't know
showcase the world I don't know that's
something if your business and you want
to keep track of your competitors maybe
that's a very specific task that no
other company is doing right I'm this
kind of general whatever information you
need and it would be too tedious for you
to like do it by hand maybe I want to
like have like a dashboard page which
shows like my five top websites maybe
they make their information publicly
available yeah so I'm happening like a
parser that puts it all together in one
page maybe that'd be compelling yeah you
can have your like own curated news feed
or so yeah yeah do something like that
actually with transit screen I don't
know sure if they is the same I think
they use api's for that but the same
kind of idea yeah yeah so anymore so web
scraping is not the only way to
selective for speaking of API yeah
speaking of api's uh generally web
scraping is kind of like I would say the
happiest way to take information because
it's not really like official you can
really just do it based on an its HTML
page that is given to you
you load a website but before you go
straight to web scraping there are much
easier ways to access information the
first of which is an API which stands
for application program interfaces I
don't know have you guys had a web
stream on API we haven't done a web
stream on API yet I don't think yeah I
don't think we've touched I think and a
couple streams think we might have
briefly mentioned what they are I really
got too much detail yeah so just
generally speaking usually larger
companies or organizations will actually
create their own libraries and set of
functions for a user to use to easily
use their services I think the most
common example is maybe Google like a
lot of people are familiar with Google
Maps if you want to integrate their map
on say your blog or something you don't
need to reinvent that wheel or like
scrape their maps they have like a bunch
of nice pre-made functions that you can
use that's gonna be quite easy to others
and I often put this sort of like behind
a like a payment structure this is
exactly you know yeah use it for free up
to a certain amount of requests right
where we ask their server for the
information but then past maybe like ten
thousand in a certain period of time
they're like no you have to start paying
for the server yeah yeah sure we have a
couple of new followers as well I'm just
gonna keep it on this page I think here
which was Elia not a cast in cyber
videos thank you very much for the
follows yeah so that's kind of the most
official way to interact with another
company's information because that
having an API kind of signals from the
organization that yes we do want you to
access our data or maybe use our
services and here's officially how we
want you to do it right so that is kind
of like the safest option if they if
that is available definitely use that
and you'll get obviously maybe some kind
of customer support in that case too
yeah yeah you kind of just get the whole
suite of whatever the organization grace
wanting to give you um the other thing
is publicly downloadable files like if
there's files if data already available
one example I can think of as maybe your
city or county has statistics for your
area that could be interesting to
analyze I'm no need to like force it
when you take that from them that
usually have maybe like take a CSV sort
of giving you the database upfront
exactly yeah
yeah so these the reason why I really
want
highlight these before we move forward
is because these are really like
official notifications of like this is
how we want you to interact with us I'm
so always looking out for those
but if maybe they're a smaller
organization maybe a personal blog or
something where they don't have the
resources to necessarily put together an
API or something web scraping is a way
for you to kind of like take initiative
and say like hey I'm gonna structure
this how I want and kind of build things
from scratch and even even bigger
websites to some big websites to keep
everything sort of at least you know the
API why is it you don't really expose
one and I think Wikipedia I'm guessing
doesn't have an API I actually haven't
looked it's probably plenty of websites
even even if they do they're probably
plenty of websites that do have a ton of
database a ton of data and Aundre in the
chat is actually pointing out IMDB
doesn't have a public API that is
certain times where you kind of have to
do it yourself
right do the happier way of getting the
information yeah and I think that that
kind of falls into two cases like one
maybe the organization very
intentionally doesn't want to make it
easy for you to access the information
and I was like playing with
beautifulsoup a little and I was trying
to scrape something and I got an HTML
page that I didn't recognize
so I was like what is this and I said
something like BOTS not allowed okay
okay servers do have a little bit of
protection against ideally anticipate
this problem also harsh gee okra and
lasts ages right thank you very much for
the follows yeah so that's kind of one
case the other case maybe they don't
have the resources or just haven't gone
to that all right yeah well I'm very
excited to see a little bit of what
we're doing with your beautiful soup it
looks like what is beautiful see we
talked about this a little it's
basically a Python library that is able
to parse and organize data from an HTML
or XML file we're gonna be focusing on
HTML files today and HTML files again
are just the markup language that
today's websites are written in it
represents kind of the structuring
outside of maybe the colors and the font
size and whatever but just kind of the
real content of each website okay
awesome
yeah so what does it do it parses it and
breaks it down into a tree of job Jex
right and is this perhaps related to the
Dom that yes your pages are
sort of constructed by and yeah so Wow
amazing you got a flow here yeah so if
you ever use HTML or you haven't that's
totally fine basically things are
organized and tags where the tags label
what kind of information you're storing
between them okay so at the highest
level like all HTML files are like a
document to just signal like hey this is
my website and then from there it kind
of splits into different things
depending on what you want on your
website so each tag is kind of like a
tree node almost yeah like yeah for HTML
it's I think of it more of a label and
then I think of the Dom as actually like
translating that into the tree okay
but yeah so you can think of maybe I
want a title on my website maybe I want
some text and a link or whatever these
are all denoted by tags my HTML
oh sure cuz like a an e/m tag for
example that's not a doe that's a style
attributes and that no yeah there's not
a one-to-one necessarily mapping of text
to nose I see what you mean
yeah so kind of the general idea of hTML
is it's a markup it's just telling your
browser which is the thing that is
actually making your website look like
what it's looking like what each item is
so h1 for example stands for heading
line script is actually not like
something you would see but that's a
JavaScript script title that's actually
in the head so that actually doesn't
show up either but that's something like
a P tag that would be a paragraph text
there's h1 through I think h6 I believe
just different styling and the world
might think that you actually made this
diagram yourself because they they don't
see the information at the bottom
unfortunately but disclaimer I did not
make any images unfortunately but it is
very it is very lovely looking though
very helpful
okay so let's talk a little bit more
about the object the tree thing that
you're talking about so HTML has kind of
all these tags so your browser can
render a certain way like I was saying
but what beautifulsoup does and other
HTML parsing interpreting libraries
slash programs do is they say hey you
know how you structure HTML is very
similar to something what we call a tree
and that tree can be like you can move
around and manipulate that tree to do
different things and so I guess like
what exactly is a tree a tree
is this kind of structure you have each
of these boxes would be called a node
and you have various relationships
between these nodes either they're like
children of or descendants of or they're
like siblings and so I think the most
common example is like a family tree
maybe people are more familiar with that
so maybe this is like the great-great
grandpa or whatever and this is like the
next generation the next generation
these are siblings because they're
horizontally even and same thing here
these are siblings maybe they share the
same parent and so on and so basically
what beautifulsoup does is it takes this
markup language that was only used to
maybe like render a certain look and it
actually breaks that down into different
objects which objects are just a way a
fancy way to code such that there are
certain features that you could access
very conveniently now typically I think
most people are used to seeing trees
kind of like flipped but programmers
tend to represent their trees as being
like top-down yeah yeah I didn't even
think about that it does tend to get
wider as you go down yeah yeah yeah yeah
okay so kind of moving on I mentioned
this idea of breaking down HTML into a
tree of objects and so now more
specifically into like what
beautifulsoup does with those objects
first and foremost is the tag that's
kind of the like it usually identifies
each node of your tree in HTML so attack
could be an a tag a P tag h1 tag
anything so these maps sort of
one-to-one to the HTML yes yes navigable
string is its fancy word but it's
basically a string so say it is
navigable that's some feature is that
beautiful soup added but basically
saying you know like a P tag and the
text is like hello the string object
would be that string hello
so it's not strictly a tag in the sense
that it's like not a link or something
but specifically the text inside the
link okay and we'll go a little more
into that the beautifulsoup object is
what they call the like overall object
of your whole HTML page and so we'll see
what that looks like but basically
that's like maybe the very top node in
our object tree in a comment is
basically if there's a comment in your
HTML that doesn't usually render on the
screen but it's gonna be in your file so
it's a way to parse stuff oh that's cool
so you can even store like men
in comments then and parse that with
beautiful suit yeah okay probably most
people probably don't design their
webpages that way but you can
theoretically do it you're good yeah
yeah I'm a fan of the conciseness and
cleanliness of the slides by the way
yeah so we talked a little bit of tags
they corresponds pretty much one to one
with HTML tags and they use the same
identifier so P tag and beautifulsoup is
the same as a P tag in HTML and
basically the object part of these is
instead of just representing a tag in
HTML you can add like style attributes
you can add like maybe IDs and classes
beautiful suit basically encapsulates
that into one object and allows you to
access that information by using the dot
operator which we'll go into again later
so it's essentially a dictionary that
holds what kind of tag it is but also
some of the other information ok cool
sounds very robust mm-hmm so here's an
example here in the top line here we're
using a beautiful soup constructor and
kind of America show it really quick
where I'm getting all of this
information is from the beautiful soup
documentation and so this is how did I
get here I just googled beautiful soup
and this is like the second thing that
came up I can actually probably paste
that in the chat so if I knew beautiful
soup dogs and then go over here or are
we using for point 4.0 I guess Roberts
doesn't matter too much but okay yeah
I'm gonna paste that into the chat and
then X polar dreams is asking is there a
link for the slides you haven't have a
link to the slides well actually do can
I should I show that now you can show
that you can send that to me by email
and then I'll post it in the chat okay
right um and then I can actually all
switch to uh all right temporary page
here if you wanna your screens not
visible now so if you want to send me an
email I also real quick open everybody
who wants to see the slides can see them
and also Astro tea thank you very much
for the follow some was saying that I
wear a specific striped shirt often I
kind of do actually Andre was saying
that web scripting is an illegal per se
just hacky yeah and the sensitive being
hacked on the sense of it being hacking
and I am Watney thank you very much for
the follow as well and if anybody has
any questions and what we're doing as we
go along definitely ask them the chat is
currently hidden just because the oh it
looks like I brought them back up onto
the screen oh no because it's a
different view but I had hidden the chat
briefly just because I was taking up a
large part of the slide deck but sorry
oh did you save me the slides so one
second everybody and we get the I don't
see them l see it just yet I might be
having a slight delay but we can go
ahead and continue here I'll go ahead
and bring us back to your screen there
all right yeah so I was this is not what
I wanted documentation basically these
are I assume the creators of this
library and they're creating this nifty
guide to help us use their product in
case this library okay so yeah kind of
I changed the form of the document
documentation a little bit but here
before you can parse anything you
essentially have to create a beautiful
soup object right kind of how this
library is gonna take in the information
and do all these convenient things for
you we did Brian did do a nice little
stream on object oriented programming
and so this kind of ties into that where
there's some object the library creators
have yeah isn't it it should be a
beautiful soup thing that she does stuff
and stores information exactly yeah so
we've made a beautiful soup object and
we've called it too haha so this object
is actually just one HTML tag with it's
a bold tag and the text is extremely
bold and so I set that tag to be so I
took the soup which is here and I parsed
I said give me particularly the B tag
which is happens to be the only tag in
there and I set it equal to this
variable tag um so now if I do tag name
I should receive the B tag so name being
the name of the tag and if I want the ID
of this tag and so this is where you
kind of seen the dictionary coming in
where ID is the idea of the value for
this dictionary that's little confusing
right makes it nicely sort of indexable
for us exactly like if you're familiar
with an array you should
that's my number but in a dictionary you
can index buy anything you want so here
we have ID and it's boldest as we can
see here and then if we get attributes
it'll actually return us a set or I
think a dictionary of all of the
attributes so maybe if we had more tags
maybe a class or something it would
actually return all of those things
right thanks to Caesar 4110 and
PewDiePie one for the follow so a little
more about the navigable string it's
basically a normal Python string except
it's got some added searching
functionality beautifulsoup is the
entire document as I mentioned earlier
and a comment is just a normal comment
so I'm not too much there ok pretty cool
so this is this looks like it's getting
into the sort of the meat and potatoes
of what we yeah isn't what we do yeah so
if we kind of understand this idea of
every node of the HTML object of the
HTML page as being stored as some kind
of object with some fancy searching
private properties and indexing
properties that beautifulsoup has
created for us they've also found a way
to kind of link all these pieces
together to really form that tree and
I'm next I'm gonna be going through some
functions to be able to navigate that
tree and maybe if you want a piece of
information like the last heading or
something how do I get that and even
maybe to find a specific piece of text
maybe you're searching on maybe you know
the area that you're looking for says
like schedule for example yeah you
probably I'm guessing it probably looks
for it in your document yeah hence the
navigable string right searchable yes
yeah so it's interesting to have you
know how to parse one tag but more often
than not an HTML page is gonna have many
tags and so this is how to get to what
you need right so first we're gonna look
at navigating the tree so here's our
beautiful tree diagram that I did not
create Oh again nobody knows though
because this is so this is just a little
more complicated than when we saw before
we have our main HTML document and then
these different kind of like tags or
categories within the HTML page so maybe
an image heading paragraph I do all of
these things are accessible ok for
before and thank you very much for the
follow um so yeah generically ever you
talk about this but
beautiful suit provides functions to be
able to move along this tree in a
designated way okay so here's an HTML
page example looks kind of complicated
it can honestly barely fit it on this
slide but just to give you some
perspective this is the HTML page oh I'm
so very very simple the code looks kind
of complicated you can imagine that like
I don't know Google's Google isn't even
that fancy like I don't know Facebook
page it has a lot more HTML tags than
this right
but yeah so genetically this is a very
simple HTML page we can see that this is
the head this is actually kind of like
meta information that is not doesn't
display directly on the screen where you
typically see your website but the body
is kind of the stuff that you actually
see so we have a title under this P
class we have it we've given it a class
title this B tag just it's bold
here's another P class and then as you
can see it's not just that the tags
surround content but tags can be fit in
other tags right and so that's kind of
where we get this tree child notion so a
good example of that is here this P
class this P tag with a class story
that's one big chunk of paragraph text
essentially that I just needed and then
within it there's these a tags and a
times in HTML are usually links so this
is saying within my paragraph text I
actually have three links and this is
what there was links look like you like
the P is like a note here and then
branching off you get the three a tags
yeah so there's the app there's the
three kind of like this and then a
sibling to this P class would be maybe
this P class but it doesn't really have
anything right now right and then the
whole page ends those would be three
ptex coming off with a body node so
those are three that's one of the ptex
has the three coming off of it so you
get like essentially the tree structure
you showed earlier yeah and that tree
structure just like I said it gets more
and more complicated as you make things
right yeah so yeah very simple HTML page
so how can we kind of access these
things we have a sorry actually can you
undo the crop it's an eye I can read I
can undo the crop yeah it's probably
more hard but basically like we did
before with the beet
we can access certain things so say
we've already made the soup and we
called it the soup variable we can get
the head by saying soup head and this is
just generally how you access
information through objects we use this
like dot operator
so we're saying I want the head of the
object or of the soup and it returns to
me this which is exactly what I see here
and then say I want this be tagged but I
can't directly see that from the very
high level soup so I have to designate
okay first I need to get the body which
is kind of the first level of things
that I see so this body tag here is like
on the outside and then within that body
I want the first B tag so this dot
operator is you can think of it as like
going step by step I'm saying get the
soup get the body and then within the
body get the B tag you're like nesting
in indexing it to the object yeah and so
basically that will return the B tag and
then Adam had asked is the navigable
string part of Python or a beautiful
soup that is a beautiful soup thing so
it's particularly for their tree
structure and how to navigate it is
helpful yeah okay so that's using tags
another way is you can use contents and
so contents is just kind of a way of
saying like what's inside this thing and
as we saw before there can be many
things inside one thing so here we have
the head tag and we're gonna say stupid
we knew that return the head tag before
let's assign it to a new variable just
for convenience and then if I like print
that it's gonna tell me that that's the
head tag which is what we saw before and
now here we're gonna use the content dot
operator and it's gonna tell me within
the head tag there was one title tag and
within that there's actually a string
that's just the Dormouse nice so we can
pretty easily get to whatever a specific
piece of information we want yeah we
kind of do have to know the layout of
the page there exactly but not only that
but using like print statements and
various things to like reveal things to
yourself you can actually determine like
say I don't know what's in the head tag
if I use this content operator it
doesn't require me to know anything
about the page ahead of time
it's gonna actually return what was in
there okay oh yeah expo dreams asking
contents returns a list it does return a
list that's very astute so this is a
list of length one say I wanted the
contents of this P class there are a few
things in there so I'm I think it would
return like a list of length three with
all those things okay make sense would
it give you that text as well that once
upon a time yeah so maybe it would be
five because there's like text and then
an a tag a tag a tag in the text again
right I'm not completely sure off the
top my head but it will give you those
contents something that our viewers
could definitely try on their own
beautifulsoup 4.0 yeah I missed some of
the stuff how do I use Beauty how do you
how does one use beautifulsoup how does
one use it on a website yeah so
beautiful soup is a library that is
available for anyone to use how you
would use it is there installation
installation instructions on the
documentation you should be able to use
pip or the Python package manager to
download that and then you use a simple
import statement to use it in whatever
code and then you can't actually use it
like I'm a little confused but what you
mean by it how do I use it on a website
but basically if you want to parse that
websites HTML page you could either like
manually download it and do something
with it or you can use Python to get
kind of like HTML request library or
HTTP request sorry library and we can
talk about that little log and we should
clarify this is meant not to be used in
an actual website that you deploy but
rather if you're running a script on
your local machine or on a server or
something that it's a back-end sort of
use case not got to be deployed in your
actual web page yes it's for analyzing
other web pages yeah yeah maybe if you
wanted to make like a web app that does
that yeah theoretically do that - yeah
yeah yeah but typically probably on the
back end which she said thank you very
much for the clarification and also
thank you to peril cakes and CL prism
for the or CI prism follows okay so
that's kind of the content tag here
specifically say we only wanted the
first item in content as someone a
student you notice that it is a list so
that specifies that we just want the
title and then here within the contents
of the
title tag is actually the text of the
text of the title itself okay oh there
we go I think I must just wait for your
mouse to stop oh yeah I think anyway
okay so there's also this idea of
descendants and this ties back into the
idea of the whole HTML page is a tree
and you can go parents children siblings
just like a family tree
and so descendants is saying everything
below the head tag like what's inside
there um so we've identified the head
tag it's the same dormitory that we were
talking about we're gonna say a simple
for loop head tag descendants is a list
so it's an iterable it's something we
can use a for loop for and we're gonna
print every item in that list so if you
can see here it's just the title and
then actually the content within that
title so there are two items that are
descendants of the head as virtue of it
being an iterable super easy to get all
the information yeah it makes it nice
and easy
yeah and Python as jobs not mobs and
free step they grow much for photos okay
so you can also use string and that will
return actually just a unicode string of
whatever's inside and I think this is
the navigable string thing so this
allows you to search for that specific
object right very similar just using the
dot operator very nice
okay so generally we have directions of
moving up and down the tree contents and
children being moving down and parent
and parents very intuitively being
moving up also we have ways to move
horizontally in the tree so like my
siblings exactly you feel like there's a
tags if you want the next sibling or
next siblings the only difference
between the singular and plural here is
one is gonna return a single object and
the other one it's going to return a
list of objects depending on how many
there are
okay so yeah very similar just the other
direction previous and previous siblings
previous elements so that's kind of if
you want to if you think of yourself as
like I don't know a bug walking through
the tree and you can only know
directions that's kind of what these
functions to but oftentimes you know
exactly what you want and you can just
say like I see that one like give me
that one and that's where we get into
sir
so element would be more for like in the
case where you want to differentiate
between like a sibling element versus
maybe like a styling element or styling
tag rather yeah I'm actually not a
hundred percent sure I think these might
be equivalent and like just like
different syntax but we should look into
that and that's definitely something you
can check on the documentation for cool
specificity yeah okay so searching the
tree like maybe I identified something
that I wanted I don't want to bother
like filtering through everything to get
there I'm so sorry I specifically want
the title I can use this find all
function and the quotes are accessed by
class and so I don't know if you guys
have talked about in the strange but
like IDs and classes and like I did yeah
I did a couple streams HTML and CSS we
went over the basics sort of yeah yeah
yeah so a class again it's just a way
usually it's for like adding specific
CSS but it kind of serves a double
purpose here as an identifier right yeah
so here we're saying we specifically
want things with the class title and
then it's returning as it should and
then we can use multiple attributes
actually so what if we want P tags and P
tags with the class title and it's gonna
give us that specific thing Amit's was
asking can we make a website and app
without coding and certainly I think
there are two graphical tools for doing
this I mean I think even scratch is kind
of an example of that I mean it's not a
website or an app but it's graphical
it's graphical programming but there are
tools like Wix and WordPress that you
can use to develop websites without
needing to code anything yeah apps I'm
it's been a long time since I've looked
at like the like Xcode and like iOS
development but I think you can do
certain amounts of things without
needing to code right yeah I'm not sure
I think maybe for like UI there are
easier ways to just create an interface
but logic wise I think really those like
learning language is it yes I know I
know for like like if you're doing unity
and unreal game development which is
more where I have knowledge and unreal
for example they have a graphical
programming language for a slore to
scratch but is robust like you can do
anything with it basically they're
called blueprints unity has the
equivalent type thing and
imagine they're probably similar
technologies that exist for other
domains like web and app development but
I'm not entirely sure I don't have much
expertise in that area but some people
in the chat are saying yes that is
possible Weebly and Wix says whip streak
who sort of to what I mentioned yeah
WordPress etc etc more ways to search
the tree we can sorry I misspoke that
the quotes are actually just for tags so
we're looking for tags here and because
I'm using final it's actually gonna
return a list with everything so here we
have the three links that we saw earlier
nice um so again using find all this is
all a tagged and B tags are just kind of
like a double I don't know search here
we can actually search particularly by
ID or class and uses pythons it looks
like named argument to do that which is
quite cool so this is telling the find
function I want the ID of link one
versus before if we just have like a
generic quote it's gonna assume that's a
tag so here we're getting this specific
I think is this one yeah link one
because this has an ID of that and so
it's gonna return that object so that's
using attributes and also so if you
think searching by ID or class or tag is
too restrictive like maybe you want this
very specific thing word like the word
is this and I don't know all these
specificities do not fear there are ways
you can do that a good thing about code
is everything is super customizable so
you can actually create your own
function using your using the functions
and beautifulsoup but using some logic
to maybe make it more complicated and so
this function is called has class but no
ID basically I want items that have a
class but no ID so I'm using these are
beautiful soup functions here using some
logic here and then now I can use my
custom function to actually search for
things so you're basically writing a
filter sort of yes and using it as a
higher-order function or I guess a
first-class function you're passing in
to find all
yeah and find all will basically call
this function for every element as it's
looking through the tree yes that's
pretty cool yes quite neat and also
thank you to very well and cell
58 2019 and DJ Moreau key for the
follows appreciate it you can do more
stuff searching particularly like
putting things together italic and this
class that kind of thing
search by string memory string that's
why we have this like navigable string
thing or else your beautiful soup
wouldn't know that that string is
whatever that is right I'm so that's
kind of neat you can also say it's a
huge webpage and there's like 25 a tags
you're like oh I just want two of them
well there's another parameter called
limit which is very very efficient I'm
assuming yes
so just generically you can find maybe
instead of looking through the whole
document which maybe would take more
time than you would want you can say
specifically within siblings I want to
look for this particular on or whatever
kind of comparing or combining the
navigating and searching smells like CSS
sort of Oh a little bit yeah using
select select and find are very similar
I think that the syntax of the arguments
is just slightly different um so like we
want three of the P tags and we want
these particular tags and so on yeah so
different functions basically to pick
out pieces of the soup that you want it
looks like X polar James s can you grab
any attribute can you grab any attribute
I think so yeah so just give you all the
attributes that the tag has right just
look for the specific one you want yeah
if it exists it should be there yeah
okay awesome
um FIFO says hi what's better scrapey or
a beautiful soup yeah I'm not super
familiar with scraping I think that is
one of the more heavy-duty was I think
so I think it's more for like recursive
like web indexing web crawling this is
more for like HTML parsing yeah it smell
parsing like just lighter use maybe you
want to look at one website or something
right yeah so probably similar like
concepts and how they work okay yeah so
what can you do at the tree we kind of
talked about observing it and like
taking information for it but how did we
started with like oh make the soup
beautiful so there's actually some
functions that I think we're not going
to go into too much today that you can
say add styling to or like in a
programmatic ways maybe I want to take
all the a tags and change their font
I could do that in a very efficient way
same thing modifying the tree maybe I
wanted to add a new paragraph or move
this paragraph somewhere else a
beautiful soup also enables you to do
that yeah the next question is like okay
now I have the tools to get all this
information like what am I gonna do with
it um so it would not gonna do with it
it's oh yeah well that's a different
question so a few things you can output
your findings maybe you're doing an
analysis on all websites that talk about
the Panda I don't know are we speaking
from personal experience I have not done
that no but maybe a personal project in
the future yeah if you're doing a
research report or something you could
write it to a CSV or a comma separated
values file or a database to manipulate
with some other tools create your own
website like say your sprucing up
someone's website or there's just some
information that you want to put
together combine that with your web
development knowledge and some kind of
data analysis if you want to so really
what do you do with this any this
information well like anything you want
with any other information this is kind
of really a method to get certain
information programmatic right it looks
like Adam asked could you use this to
make a search engine like Google ah so
Google does use some kind of oh
definitely not beautiful super like
technology to parse certain websites
that being said their primary technology
I would say is how they rank their pages
and how they decide to that a certain
website is more relevant than another
you could imagine that if you created
some technology to just identify all of
the websites and then return it and
whatever list you got them that's
probably not going to be so helpful as
saying Google is great but yeah I
imagine they do they have to do some
kind of analysis on web pages to be able
to hand it back to you
so 5820 19 asked is the documentation on
the beautifulsoup website they didn't
resource to continue learning I would
say so yes I I read through it and I
thought it was quite intuitive
oftentimes documentation can be not
great and I thought beautifulsoup was
one of the better ones that being said
there I think third-party like somebody
else has
the documentation in britainís maybe a
more intuitive guide or maybe videos
even and I think certainly doing like
projects using it and they're like other
people's projects probably at least it
might fill you in on the why exactly
yeah yeah or some inspiration if you
need it
boy oh boy thank you very much for
following as well okay so we talked
about how scraping is kind of like
hacking right like it's not super
official that's kind of your own
decision to do this but there are also a
few things that could prevent you from
doing all the squirming that you would
want a very different thing so bad like
formatted HTML code this is actually
huge because the whole idea with
beautifulsoup or with HTML is that you
assume whoever made it was kind of
logical in how they made it so maybe the
title is in the body and within like the
P tag is the beat action letter but say
like there is like a random generator or
like a child maybe HTML head and it's
just disgusting and nothing makes sense
that is possible but it will definitely
take you more time it's funny because
one of our viewers Nate is 10 years old
and makes HTML pages and actually we
critiqued his page it was actually very
great I mean I would expect to you know
someone who's very young to not have
that much experience but at least we did
an awesome job that's really cool I
think he's definitely an outlier though
in my experience so far yeah and just to
clarify when I mean badly formatted I
don't necessarily mean that like oh it
doesn't render like the tags are
unwashed or something but kind of
irregularity so say I have multiple
paragraphs and for some of the
paragraphs the link is outside the
paragraph tag but like for some of the
other paradox the link is inside the
paragraph tag so that kind of like I
can't expect what's coming next right
that is what makes it difficult to
scrape yeah having things in conceiving
on the same website like having and we
were messing with this before stream
having one article that has you know you
assume based on analyzing a couple
articles and they have a consistent
schema yeah then you choose a random one
and it happens to differ from that sort
of non-deterministic Lee or unexpectedly
it's kind of like I don't know what
metrics to use to it
mm-hmm so that's a limitation in like
it's technically difficult to deal with
another more if like the website doesn't
want you to be there that's kind of a
different purpose authentication systems
say it's like we were talking about
email you can't just scrape someone
else's email INBOX like there's some
kind of password or something protecting
that yeah
session based systems if there's certain
information that's actually stored on
somebody's cookies or like locally on
that computer that's not public
information for you to access so you
can't scrape that like I mentioned
earlier when I was trying to scrape I
think it was like I don't know
hotels.com or something but some
websites will actually identify when a
user has made more requests than they
can handle or that they want to deal
with like and they will actually block
you and they'll say like oh we know
you're a bot like stop and they won't
let you access their information right I
feel like that's probably one of the
pretty common things especially as we
see in the future right
yeah I feel like more people are
probably gonna start getting hands on
this like training like deep web mining
and stuff like that yeah especially as
like like more information is like an
advantage yeah but yeah bulk access and
that actually leaves two leads to DDoS I
think is the acronym but stands for a
distributed denial of service yeah and
actually there are some like I was
serving it's none of you but some people
who just make a lot of requests to a
website like faster than any human can
obviously with a bot in order to shut
their server down and basically not
allow anyone else to access their server
so that's actually like like a malicious
attack using this kind of HTTP I'm
pretty sure David himself probably did
that for it because his PhD was on
botnets oh that's a David if you're in
the chat confirm or deny whether you
yourself have orchestrated any DDoS
attacks for academic purposes we would
be curious to know for academic I'm
kidding but also there are like legal
barriers especially as cool to mention
like as information becomes more
valuable and maybe privacy is starting
to like people are more aware of it
there are new laws about how data is
shared so one thing I actually looked up
is that I didn't like copyright I think
most people know about but there's
something called database rights that I
think it was
I'm not completely sure but I think more
common in Europe and we especially the
idea that there is some kind of creative
work going on when somebody creates a
database so if you just copy someone's
database that that is like infringing
upon their work interesting cuz I never
heard about database resum no me neither
I was searching Wikipedia so it is
coming a common theme in today's
discussion okay do all websites have
robots.txt file ah yes um so not all
websites but more than I thought would
yeah that's so kind of what robots.txt
is it's a protocol that some websites
put in order to just tell Bob's like hey
we think it's okay for you to scrape
this but not okay to scrape this and
that kind of thing it's just a way for
the website to give a message to
whoever's trying to use their data so
what that is is like if you go to the
home domain like a single flash at the
end of the URL and you type robots that
tasty so actually we can try that what's
what's a website I was looking at Google
so I'm on Google right now and if I go
to google.com and I do robots.txt and
this is I kind of when I discover this I
kind of felt like it was an easter egg
that I didn't know was there but husband
here all along so this is Google's
robots.txt and it's basically their
protocol or set of rules for what they
allow BOTS to access and not access
interesting so if we it's saying like
you're not allowed to access the user
agent star star and computer science
more than more often than not means all
you're not allowed to it's a search or
scrape anything with slash search but
you can do slash search about and so on
and so on and Google is a very large
website so they have a lot of stuff
going on right yeah but even like
Harvard like I don't honestly think of
them as a very high tech web site but
they hover robots of to see which yeah I
didn't expect so a lot of similar thing
that what they allow and disallow allow
star yeah I hope no the hosts are so
like crawl delay this is kind of
interesting too I'm not exactly sure
like what unit this means but
essentially that they don't want you to
access their page like so often like
maybe they're protecting something I
would assume Melissa
yeah okay yeah that make sense but yeah
okay so little small derailing but that
is robots at a cheap but not 8/10
suggest some tutorials and resources
please for a beautiful soup do you know
how many tutorials besides documentation
I can't think I'm specifically enough
baton my head but I'm sure if you google
beautiful soup some hopeful Xcode yeah
beautiful suit I would say I would say
yeah Google beautiful soup tutorial I
mean this is kind of a tutorial but the
documentation is good if you don't
understand HTML I would say probably
look at some HTML tutorials first cuz I
think this kind of assumes that you know
yes you melt it also assumes that you
know Python
yes so I think if you don't know Python
either I would say learn Python then
HTML and then start digging into
beautiful soup which kind of ties the
two together yeah a few barriers but the
good news is that if you do know Python
in HTML this shouldn't come quiet right
yeah
please plan a session for non tech folks
how they can do stuff without coding
like a session on autumn also a session
on automation so yeah we could do
something like that they featured maybe
with scratch maybe have a scratch stream
automation is such a vague general term
that can be used for anything for
industrial applications to coding so
it's hard to say what we would do and
how we would also sort of reconcile non
tech and automation together so if you
have more ideas definitely toss them in
the chat and then Adam was saying at
YouTube always has something else to
Marsh boorish and married the lip yeah
sometimes I wonder who's making those
websites but I'm so grateful oh yeah
yeah are those there's videos I meant um
people that have Mississip Rob we have
thought the same thing and they're like
damn learning tutorials all just
learning I'll make one myself so that
other people have to suffer right
humanitarian yeah so these are kind of
technical limitations to what you can
scrape as we were talking about but also
like is web scraping always the right
thing to do and I think this is
something that more recent than before
has come up in like people's probably
consciousness and what decisions that
engineers make yeah so here's one
example a government site containing
public data offers no API to access the
data but nothing on the site indicates
that
agency operating it has a problem with
scraping so we talked a little bit about
like an ACI is an official indicator of
like hey we want you to use our
information right but in this case maybe
they don't have it I think like my
personal view is it's a government site
the government is usually not super -
not technologically advanced so I would
say that this is fine just kind of like
there's nothing that says they don't
want you to share it maybe if you think
that like using your personal discussion
not the information on that site isn't
particularly sensitive just kind of all
of those judgment calls
right yeah another scenario like maybe a
private site that has private data was
assembled through the hard work of some
of its owners and offers an API but you
don't want to pay and so you decide
you're just going to scrape and this is
doable because just a little bit about
web development but basically how you
pull up a web site is to pull some code
files to your computer maybe it's an
HTML javascript file that kind of stuff
and your browser so like Chrome Safari
Firefox something like that will render
that to show a certain picture so the
information if you're seeing it on your
page is actually local on your computer
so no matter like as long as it's not in
their servers you can do whatever you
want with the information on your
computer the question then becomes
should you do things with the
information on your computer so yeah I
had I had talked about this concept with
some people over their summer and I
remember some people were saying oh you
know like somebody were saying like on
private data like I probably shouldn't
look at it and then another person was
saying like oh you know it sure it's a
private site but I'm accessing its
accessing it online and there's still
sense of like publicness when you post
something online and they should know
that by the time they post it so again
these are things to talk think about
like beyond the tech side of things and
just kind of like how you should decide
your action sort of a moral discussion
yeah john 8 asked can you do an intro on
cic d IE Travis or Circle C I please we
actually did with Kareem so you go to
youtube.com/scishow you go to our twitch
playlist you'll see it was one of the
earlier episodes we did
Travis CI continuous integration
continuous deployment stream so I can
check that one out
and then someone actually kindly pointed
out Marie Dilip said already cs50 has
authority over Travis CI on youtube yeah
so kind of what are some guidelines to
when to scrape and when not to scrape
again API is using official like
declarations when possible so if that's
available respecting their terms and
conditions I know nobody reads this but
maybe if you like scroll through and see
something about like data information we
could use a web driver to scrape through
the tree oh yeah yeah do some language
processing midair respect robots that
txt that's something that I didn't know
existed until I did a little bit of
research on this so yeah take a look at
that what it says being gentle not
making too many requests at once again
that's the DDoS as I hear saying yeah it
would say details yet DDoS that might
harm like smaller businesses or
something
identify yourself don't like say like be
transparent about your knees what you're
planning to do with it only take data
that you need for whatever project
you're working on respect the things
that other people have created like I
think often on the internet it's common
to feel like it's just a web page it's
anonymize like it's just there for my
you stand however I want but like
somebody put that up so I'm going to
like put the work to put that together
and so just being respectful of that and
also like create something constructive
like technology I think is just use best
when you're creating something that
brings new value to the people around
you or I don't know in the world and so
maybe like grandma put up a website of
recipes and they look really bad and
you're like I want grandma's recipes to
be seen by more people I just make it
prettier personal anecdote I know my
grandma does not know how to say that
were the case there are ways to use this
technology to actually create something
positive but again they wouldn't be able
to you would need to give this
information back to the to the server of
the information right because you
parsing it and adding new stuff that's
only gonna be on your in your Python
intern
yes yes so you would need to say you
wanted to make a new website you can use
this to like alter the HTML but that is
just gonna be text on your page and you
would have to serve that on your own or
something about nature yeah yeah okay so
now oh we do have a URL where yeah this
is oh is that and that's an IDE link
right no actually oh okay cool let me go
ahead and plug that tiny URL no it's
accessible anymore yeah I don't know if
this is shared with everyone except on
my personal Google Drive I have access
to it okay yeah I think it's I think it
works yeah so everyone who wants to play
along with it looks like somebody
already tested it exploit dreams is it
works Oh amazing okay cool Adam was
saying stream ideas algorithm analysis
and discrete math and networks those are
good ideas actually it's not about some
of it I thought about networks and
discrete math algorithm analysis it'd be
cool - I'm not an expert on any three
all three of those ideas actually but we
can maybe find somebody who is
but this here this is the cs50 IDE looks
at using the legacy version - yeah I was
the new version the last few months I
think Korean cashed out version 4 or at
least we have been pushing out version
for the new beta version but this I
really just learned this too because I
don't use the IDE too often myself we
also have now the cs50 sandbox which
people can pull up this library I'm
assuming they could probably if they
will go to cs50 sandbox i/o they could
probably go our sweet 50 that is sorry I
have that backwards
they could go to that and could probably
just install beautifulsoup yeah I don't
know if it's already in the environment
so if you I'm pulling up one on my local
machine here just as an experiment this
is kind of what it looks like I'm
playing yeah so you pull up in yours you
can kind of choose whatever kind of
window you've done it more recently so
you might know the actual command and
you got to authorize it so I steal your
personal information
but no I think now I can probably just
just like pip install beautifulsoup is
it just like that or be able to do ps4
it's BS for right pip is saw ps4 oh it
looks like it's already installed okay
so it comes pre-installed with
beautifulsoup so if I type in Python 3
and I say L is equal to a beautiful soup
the constructor and then what was the
argument addicted to take a specific
take the text of the yes so you would I
was using like the get from the issue
you want to maybe do it here and try to
try it out and see if it works yes it's
good at least get started in the sandbox
yeah so should we do I kind of have one
example with the HTML locally on your
computer and one oh sure sure sure yeah
we could do that um so yeah we could do
it here I'm gonna make if you remember
this HTML document let's call it Alice
team I'll I'm just gonna create a new
file with that in it so let me find I
think I had that file somewhere so let
me just copy this and put it in the in
the sandbox solid okay so this is the
HTML page say I downloaded it from a
website and now it's locally on my
computer so if I had no internet I would
say well attacks still be able to access
this okay and so then I'm gonna create a
new script a Python script that's going
to scrape this so let's call it scrape
Alice okay so I don't have a pot file
called that yet
let me see
we'll just open it that way that's fine
okay so there's a few libraries I'm
gonna need before I get started with the
octal beautifulsoup stuff first I'm
gonna use the cs50 library which I think
I'm actually not using I'm not gonna use
but we can use it to take in user input
this is assuming that beautifulsoup is
already installed so I think in cs50
we're used to just writing these import
statements and magically we can use it
that's because the libraries are already
installed already magically installed
yes so if you're doing this locally in
their terminal window or something you
actually do need to install the code
that is written that allows you to do
this on your computer but once you do
that you can do this simple import
statement looks like David's in the chat
not too saying thanks for tuning in
everybody
David we're using the cs50 sandbox if
you haven't noticed it's a nice it's a
beautiful tool folks should use it for
beautiful soup yes beautiful tool so
many beautiful things and then this last
librarian that I just typed in is the
Python request library for making HTTP
requests okay hopefully that's installed
to actually didn't check to see that
yeah and actually for this first program
and we're not gonna use this quite yet
but so the goal for now let me see if I
can get this split on uh I don't know
how we'll just leave it like this but
I'm just kind of the basic setup of my
Python I'm gonna create this main
function just to organize things a
little bit and here's my where main is
going to be called so if you're
unfamiliar with Python don't worry too
much this is just me setting up like the
bare bones of the application or the the
script in this case okay so now I'm
going to start writing what's what my
file script is actually gonna do um the
first thing I want to do is actually
just open the Alice dot HTML as a
read-only file and I'm just gonna
comment that out for basically a Lestat
HTML as you remember I downloaded it on
my computer it's now in a file in the
same directory as I am now so I can open
it directly to get the text instead of
maybe like copy and pasting it or
something
so I'm going to do F equals open
and open is I think within the Python
standard library and it's a function
that just allows you to open files
however you want R stands for read only
I'm assigning that to a variable F and
then I'm going to assign this new
variable s to be actually what is read
out from that file okay and the next
step so now I've basically gotten the
file into my script and now I'm gonna do
something with it so soup equals
beautiful soup if you remember this is
the beautiful soup constructor that
makes all those nice convenient objects
for you and then I'm gonna use this HTML
parser and basically what that is is
take s which is like the literal text
it's this right now and parse it as you
would an HTML file so there are actually
other parsers than in beautiful soup yes
you know yeah so if you remember
beautiful soup can also be used for XML
I'm so that would be a different parser
okay but this HTML parser actually I
think is native to Python as well
correct me if I'm wrong but I think that
repeated one more time would you say
this HTML parser I think is within the
Python standard library oh maybe I'm not
sure um yeah okay so let's just see what
we have so far so I'm gonna print the
soup let's just see what happens
a little bit yeah okay so I'm just gonna
run Python to do nice ok it looks that
looks whoop so pretty compelling I've
got some stuff here it looks pretty
similar to what I have here there's
actually a function called prettify that
I think does some indenting um so I'm
just gonna of concern to use a little
bit of that yeah yeah apply that to my
soup okay looks good it looks like I use
two spaces yeah a couple spaces just to
give you an idea of how things are
nested that way things don't get like
super exaggeratedly indented with HTML
as it does yeah yeah so uh let's do an
experiment so that's that's the soup
what what do you want to access let me
look at the HTML again so
let's try and grab the different sisters
names maybe the sisters names okay so I
like that there's a couple of things in
here maybe I need to I think the fastest
way to get to the tags first um is I
think maybe through this class
yeah the sister class yeah yeah so I
think what if we do a soup dot find all
because where you want multiple oh right
yeah
things where the class is sister oh nice
okay not a hundred percent sure if it's
what work we do we do live coding like
this everything every day we always
rented a great awesome fully working
code never about yeah okay so I actually
don't remember how to grab a class
what's good about that is there is oh it
said a slightly slide or documentation I
have two options
Oh massive this is looking for IDs and
then maybe you should just grab the AJ
because those are the only a tags you
could do that well in the meantime we'll
say beautifulsoup find all class yeah
and to be fair I did three slight
curveball because this is completely
completely random completely blind
hey this is okay if you do find all the
way this is weird syntax Oh class
underscore Oh interesting
okay and it looks like you have to
specify the tag first and then the class
oh I guess if that's only if you want
the a tags that match that otherwise you
can just say class underscore equals you
know it's weird it's the exact same one
if it's cuz this is the example though
okay okay that was gonna be thrown for a
second fair that you okay so I just ran
it with the change that Colton looked up
and also weird that they chose class oh
you know why cuz come on
okay totally makes it should actually
update my site let's do that real
quickly cool something Rama
yes so I found all things with the class
sister as you can see it's outputting as
a list down here so maybe if I want
specifically those names hmm how can I
do this that's a lot of stuff so it
looks like where's the so it's after the
a tag the ake tag closes so is it like
content did you say yeah so we can try
that but the problem is now that final
is returning multiple items so I I don't
think I can get contents directly so I
guess you could do a list comprehension
yeah that's what I was thinking but
let's just try this we'll see if this
will respond to me yeah
so it's telling me you're treating a
list of items like a single item okay so
if you did maybe a list comprehension
yeah on the right except it wouldn't be
I think a list comprehension or we can
do a filter I'm just gonna do a for loop
it looks like that you've missing an S
there too OOP
let's call for s in sisters we could
also use like a more of like or
something but I want to get the contents
I think it's just content and let's just
print all about this may or may not work
live coding and it should be contents
right is it content I think it's content
okay okay oh no we got some things
instead of content it gives you a list
for each one though which is which is
interesting
oh yeah I think that's maybe the return
type of contents but if we try string s
that string works nice this will get me
just so you could say you could assign
this to a list then you could say names
equals s tostring for s and sisters yeah
yeah yeah so names s top string let's
try that and then you have to put it in
a square brackets two to make it a list
comprehension which this yeah do that
yeah okay let's try that out
why didn't you tell me so yeah that
would make sense just so I can see
what's going on nice there we go we calm
mission accomplished so maybe if it was
a really big family and I had like 2,000
sisters this would be super helpful yeah
yeah oh yeah so that's kind of a small
example of some of the stuff that we
were talking about on the slides we can
move to a slightly different example
with I think I was thinking of doing
some stuff with Wikipedia oh yes let's
do that
yeah so let's call this we may or may
not have done a little bit of testing
before the stream on this yeah and we'll
see if it works
we're gonna just making a new file
called mini wiki pie and the reason I
chose Wikipedia is one it's great super
helpful the second is that I think in
comparison to some of the more widely
used websites the UI is kind of simple
um so maybe if we look up the what is
something computer science hey Pierce is
good lambda of course yeah remember
lambda some cool stuff here the format
generally is pretty similar to the other
formats of the pages it's also pretty
simple so like the text it's not too
fancy there's not too many colorings and
that kind of things the structure is
kind of simple and so kind of the way to
approach this first is one remember we
want to take whatever HTML page is being
is used to actually create this website
and we want to look at it and then using
that information we want to identify
pieces of information that we want right
um so I'm gonna say for this program I
have a few goals one I'm gonna return
the heading of whatever the Wikipedia
title is so you like this that would be
just looking at it visually I'm also
since it's like a mini Wikipedia I'm
gonna grab the first line okay I've of
the Wikipedia article so so it won't be
for example this thing that redirects
thing so I gotta watch out for that so
that might be something we need to look
around again this is with like this
isn't even an example of inconsistency
but kind of like
different things you wouldn't expect or
look out for right but I'm gonna say my
goal is to get the first sentence of the
actual article okay yep and then I also
want to get the link to the photo and
the reason why I'm saying a link is
because these photos that Wikipedia is
rendering they're actually stored in
something like Wikipedia servers or
whatever and then when you go to the
address this address here this URL
you're actually like grabbing the link
from whatever wherever they hold all
their photos so like when you
right-click and you do where is it open
open image a new tab this link here is
uploaded on Wikimedia orgs like a CDN
probably yeah that is where Wikipedia is
keeping their images so I'm just gonna
say take get that URL oh it looks like
FIFA was asking can you do an example
with a table with a table yeah so a
table as you know it's an HTML object or
like a tag as well and that is also
sleeping part so we're actually gonna do
that in this example oh nice how fitting
yeah it was like we've had natural
segues all tasted it Pyro's 2002 is
asking a really cool stream glad I came
across it thank you
unrelated when trying to cast this to a
TV from any Android twitch app I'm
getting an error unsupported source code
for interesting yeah I'm not entirely
sure Pyrus I would Google for that I
have no experience using the Android
twitch app but probably with just that
bit of code you could probably google
that and find other folks that might
have had that same issue with that app
maybe try updating the app on your phone
or your Android operating system
generally speaking if you maybe that's
out of date and see if maybe that fixes
sorry for the technical difficulties but
also go to youtube.com slash cs50 or
facebook.com cestia city if you want an
alternative place to watch the video um
yeah okay so while Colton was talking
about those two technical give
difficulties I just did some imports
various limit to what we did before
except we're actually gonna use these
two libraries this time so we're gonna
use getstring to take user input to find
whatever Wikipedia article we want
beautifulsoup and the request librarian
from Python and then just some
additional string manipulation functions
okay
so just gonna do my normal Python setup
with a main function here
and it's gonna be if okay and this will
again just call the main inflection at
the very bottom of the file that's all
it is
okay so kind of just laying out what I
want this to do I want the user to be
able to input like a something they want
to learn about input topic and then I
wanted to get the Wikipedia page and
then I want to get the title first
sentence and the image and I want to
print it out so maybe I should just say
these are kind of the things I want my
meeting wiki program to do Nate was
asking why can't we use input or raw
input instead of getstring yeah we could
get string is just a simple like
abstraction of raw input those are like
Python native functions and getstring is
something that cs50 has made for its
students in particular it's just a
little more friendly than raw I'm sort
of might be yeah but the function is
totally the same
okay so first I'm gonna handle user
input with the cs50 library it's super
simple whoop q4 I'm just gonna q4 query
and then I'm gonna pass in what I want
the user to see when they what they need
to be prompted with for their input so
I'm gonna say which PDR do Cole do you
want Adam graves Olney and buggy beetle
thank you very much for the follows
buggy doodle and then I'm going to get
that page based on whatever Q is so I'm
gonna create a new variable page and
here I'm gonna use the get function
within the Python request library and
that's gonna make an HTTP request to
whatever this URL is whoop so I'm gonna
do and no Wikipedia org slash wiki and
this is just a pattern that I found here
so I looked at this I noticed hey you
know all the pages so this again goes
into like well-structured websites if
something makes sense it makes it easy
for you as a computer scientist to like
use those patterns to get something that
you want I'm sorry I just noticed that
if slash wiki and then whatever
this thing is kind of an exception case
that I didn't realize off the top my
head but when you're searching for
things with two words you can't really
have a space in your URL because it just
there's like a percent twenty symbol it
gets like complicated with the symbols
so Wikipedia uses an underscore but I
don't want the user to have to deal with
that so I'm actually going to make a
function that puts it there for you nice
Antonio Antonio more a raw thank you
very much um so speaking of that I'm
just gonna make that function really
quick so say my user inputs like
computer space science as the thing they
want to learn about I'm going to make a
function that just adds underscores to
whatever the query is so that's not
something that's written yet but I can
make it here so it's interesting we're
kind of simulating the use of a web
browser yeah effectively yeah not that
there's a significant use for this but
it's a great learning experience and is
there a way for us to for example search
loosely on Wikipedia like if somebody
were to not type computer science
completely right or if they're to
misspell it would there be a way to like
search for a page like using an API or
something
yeah I'm not 100% sure I don't know
Wikipedia has an API I did notice that
there are some things that redirect
automatically so I'm not sure if those
are misspellings or if they're like
synonyms for the same object sure but I
did notice that that was something and
that redirect is handled here with get
will it redirect us here if they if they
do that okay yes I think it will nice I
can't think of an example off the top my
head but I think that's like if you
notice Google I think owns multiple
domains then just like goog Ellie like
right I've been like goog Ellie it'll
also read her to Google I know that's
kind of yeah clever I guess uh-huh
infinite O's infinite oh you are ELLs
yeah so I've just created my own
function add underscores basically it
does exactly what I wanted to this
replace function is from the string
library and I'm replacing all spaces and
this query with underscores so simple
does what I need it to do okay and then
I got the page from your else so maybe
that's like try let's see what we have
so far and this is something that I like
to do just debugging as you go rather
than writing out all your logic and then
real
saying that everything is wrong great
why frustrate you I'm so like
test-driven development of us a little
bit yeah so I'm gonna run mini wiki PI
it prompted me let's try a computer
science oh I did not add underscores
reference before assignment so I
basically it's telling me since python
is interpreted like downwards it's
telling me that I can't use this
function before I've actually defined it
so I think the fix for that is oh this
needs to be not within my main function
I think that's why oh yeah that will
probably do okay let's try that again
nice okay response 200 so that's not
actually giving me the text inside the
webpage but it's telling me that I was
able to access that page well 200 okay
is that's what I mean and then I think
whoops trick was asking Nate was asking
can we show the sandbox after we're done
which I guess if you do if you add the
share link up there and then give me the
URL at the end of the stream and send it
to me by a email plug it in the chat for
sure okay and then actually if you got
the link to the slides that zip at the
very end should be these files completed
oh no so but if you want to do it
together okay so now I'm able to get the
URL get that page but I want to convert
it to a beautiful super object so I can
use all that functionality that we were
talking about so I'm gonna use again the
beautiful soup constructor I'm gonna get
the contents of the page and this is
something defined by the HTML sorry I
keep mixing this up HTTP requests
library and I'm gonna parse it as an
HTML file so again let's just see what
we have so far and this is basically the
same as we got here I think it's being a
little
okay yeah so basically here when we got
to this we had to do a few additional
steps on the Wikipedia example because
we actually need to get the page from
the internet rather than having it
locally this is assuming that we have it
locally but once we've done this line
we're pretty much even so let's see as
they ask does it stop being in the main
function with just one space erased
sorry I don't know Python at all
referring to your nested yeah so if
you're familiar with maybe some other
languages you need to differentiate like
what is within what especially when
you've loops and stuff with say like
JavaScript or Java you use curly braces
with Python into white space sensitive
so what that means is the tabs actually
matter and so if you have something tab
data actually means it is nested within
something right and so earlier this def
underscore was actually within our main
function
yeah so I'm saying like how can you call
something within it like it was just
like a circular anyway it's fine now
so I printed the soup and if you noticed
my terminal kind of went crazy because
the Wikipedia page is quite long it's
little many you know there's a lot of
stuff going on even though it's I think
it's one of the simpler websites and so
another way to kind of look at this I
think looking at it in your terminal
window is kind of a headache is on
chrome and other browsers they have
something called developer tools which
allows you to kind of like dynamically
see what's going on on the page so if
you right-click there should be
something called inspect or view page
tours they kind do the same thing and if
you go to inspect it'll show you the
HTML that's actually I'm just gonna
close this window it's just okay this is
closing holding yeah nevermind let's
make this a little smaller and this is
the HTML page that is rendering on my
screen and if you notice these colors
that are kind of flashing it does this
really this is the browser handling it
it highlights different parts of the
website based on like what my mouse is
on so remember this is kind of like some
header metadata that's not really
showing up this is the start of the body
and then when I go to here
starts listing the stuff that I actually
wanted cuz remember and want the title
the first sentence and the image so
that's kind of all highlighted in there
as you can see is kind of minimized
right now so I'm gonna go in there and
see like Oh what do I actually want well
this is some random stuff it looks like
I already passed the idol
oh so the title it looks like isn't an
h1 tag okay and this is what we were
talking about like in a more complicated
website you need to figure out the
structure of their website first an h1
is actually we learned this recently
like from an accessibility standpoint
generally pages are expected I've just
won h1 for example blind people who are
trying to read webpages there will be
software in there in whatever tool
they're using that looks at the h1 as
that's sort of the title of the page and
so if you have multiple h1 tags in your
webpage it actually is not good for
accessibility
I do not know not yeah I read just
greasing limit Tom ballot or was on here
on our HTML stream yeah so maybe that's
a handy use and you could also probably
create something using web scripting to
make web pages more accessible yeah that
would be an example of like adding value
I think okay so we want to get this h1
tag it has some IDs and classes that
maybe we can use to help grab it but
okay noted that's there how do we find
everything else well there's another div
called body content but that's too broad
so let's look further these are some
different things oh it looks like this
div contains the stuff that I want I'm
gonna go and say I take a look this div
it just gets more and more specific so
I'm gonna go into there this is a note
so that's not what I want right that was
what we looked at earlier the redirect
thing yeah and luckily for me wikipedia
has actually separated those in HTML
elements so that makes it a little
easier for me oh it looks like this is
the image so maybe let's take a look at
that thumb inner it goes even smaller
and here's a table so I think someone
mentioned a table before nice Wikipedia
and displays its images within a table
so that's kind of nice because there
hasn't been a table before that so I can
actually use the fine function or
something to get the first table and it
won't anyone that I want cuz your table
table body these are just HTML
particularities the first table row it
looks like there's four photos here
aha so within the table row I don't
remember TD stands for table data I
think okay table data here's a link
actually and here's the actual image so
we see an image tag here's the source
the link that we were looking at before
and so on so that's how I would get the
photo
so that's notice that's two out of three
things that we wanted okay if we keep
going let's find where this the sentence
is so this is more tables so there's a P
tag here nice just kind of investigating
do to do these are all different links
and it looks like okay so there's some
bold this is just plain text and links
so is kind of that might be a little
harder to parse because they're split
into various type they don't vary
obviously put a class on that P tag
either to where you would know that
that's the start of the yeah so we'll
have to find a way to get to that but
there is a way obviously you could
search through the tree or a navigate
through the tree but we have to create
something to identify looks like bogey
doodle was saying WC three standards
outline a lot of accessibility stuff or
w3c awesome yeah I haven't taken a close
look at it I probably should but I
imagine a lot of that stuff is probably
very well documented yeah definitely
something to look into
okay so kind of having a better
understanding of our Wikipedia structure
we can come back to our program and now
deal with this soup and and play with it
okay it is a little bit soupy at the
moment yeah a little bit uh so like
Colton mentioned apparently the h1 tag
it's unique and so I'm gonna use that to
actually find the title so I'm just
gonna create a new variable title and
I'm going to use soup dot find which
remember we can search by attribute
class whatever we want and I'm gonna get
whoop anything with the tag h1 and then
just to be more specific I'm gonna get
them one specifically with ideas for
setting and that's probably honestly
redundant if there's only one h1 tag
right but just four and then I want
specifically so actually let's try
before we do that and say we print title
we probably eliminate the print soup to
him save us a little bit of grepping
through yes
okay so I need to enter and it looks
like i got the whole h1 tag and that is
the one that I want in awesome I'm so I
don't want all this HTML data user
probably doesn't care about that yeah so
I'm just gonna use the get text tag
although you could theoretically just
plug and place this into your into your
mock page if you wanted the Distilled
wikipedia oh yeah yeah that actually and
then you get the formatting what you
would need the other yeah yeah you would
at least get these semantics of like h
1p etc yeah so yeah for sure instead of
starting from scratch yeah okay I so now
we can see that it actually did just
output computer science as well oh nice
you had cut dot contents or a guy get
extra attack okay what dot contents work
in the same way to that we just read the
one that we did earlier oh no you did
not string right yeah so uh I don't know
let's try that but it wasn't I don't
think it was a function so so they have
set multiple ways of solving the same
problem okay so that's the title cool
what if we get the image next cuz that's
what we saw on the table okay so I'm
gonna do table I'm gonna find the table
first soup find X boiler dream is asking
what's the difference between get text
the function call and dot text and I'm
assuming means stop string illicit text
is there a dot text field I don't know
yeah okay I have to wait for this get
string maybe I should take that off for
debugging purposes but okay so there is
no dot taxes well I didn't oh you didn't
a title maybe there is then okay try it
one more time okay that works so
generally my inclination is use a
function because the function was doing
stuff underneath the hood but giving you
the same interface no matter which
version of beautiful soap you're using
but if you go dot string text they might
deprecated dot tags top string etc
you're better off using a function
instead of directly accessing it field
generally speaking
I think yeah yeah I think that's a good
way to look at it or maybe if they
decide to change something within that
function you don't have to worry about
how your application break right yeah
okay so the next step so I found the
title here I'm gonna find the image we
noted that it's in the table at each
step of kind of identifying things this
is actually a mistake that I made
earlier but I had gone under the
assumption that every Wikipedia page
would have a table and then I actually
looked up CEA's fifties page and it told
me that it didn't have a tape course
we're the ones to break the norm yeah
yeah so by application broke and I was
like oh my gosh but that is easily
solved but just putting an extra like
catch to check if the table is this or
not if they just use classes then I mean
we'd be alright right if they just use
classes it does the table not use a
class in this case cuz if they just used
an image like article header image or
sorry header or article image class then
you just always pull that you don't even
need to worry about whether something's
a table or whatever right let's just try
that let's see what happened my instinct
with this is that it's probably gonna
give us the Wikipedia logo but we'll see
if them yeah remember fine it gives you
the first whatever element on the page
so look at you that's true cs50 because
I've messed with me laughs here we go
yes canary low canary logo so that's the
trouble with you have to identify and I
think here there's not a specific like
tagged and the images in the class right
yeah all of these are specific oh wait
this is that here this is the image like
I'm not gonna know that that's a large
capital lambda before I said you know so
I think there's a little limitation oh I
see so the way that I identify it is the
primary picture will be in that table I
think that's why unless you cs50 right
unless you're sitting you have no phone
what did we and we don't have a photo
that's what it is
oh now is it the case that consistently
across any article that does have a
photo
well it'll indeed have it in the table
that has been my hypothesis and it
hasn't broken too many data six so far
suggests that this K that's the game
that isn't by no means of proof Oh got
you yeah of course um so let's go off of
that hypothesis and we're gonna find the
table and then like cs50 page doesn't
have a table so we needs to check for
that the way I'm gonna do that in Python
it's just you a simple if statement if
not table print oof you have no image
yeah and this image
variable is gonna be what I'm planning
to print later so I'm just gonna
reassign it based on what I know right
so if there's no table I know there's no
image that's maybe I don't know but
that's my guess
else if there is a table then I want to
keep looking into that table and take
out the image the image yeah yeah so
else I'm gonna do table that find the
image tag and that's gonna give me the
first one and then get SRC and remember
SRC is actually the link to the image
this is all kind of meta information so
alt is if like the image is unable to
render that's what it's gonna print all
this kind of stuff sizing and I think
that's very accessibility thing to write
like oh like localized for screen search
generators in certain modes yeah yeah
for large capital lambda somehow boys a
Linda oh would not know that that's the
picture they would have chosen but Adam
was asking another stream idea image
editing Photoshop etc
that'd be interesting it's not super
related to what we done so far but I
mean it's something that definitely
consider a more technical oriented thing
we're definitely thinking about maybe
After Effects and smaller software what
is a shortcut to make a line a comment
ah yes I so I helped teach cs50 this
past fall and a lot of kids were like
putting the the hash tag or whatever on
a Mac if you do command I think this is
forward slash at the same time that will
comment in most text editors
automatically it'll identify what the
comment symbol is for your language
um I think on Windows computers it might
be control /but yeah super handy trick
[Music]
yeah Jackalope was saying common hotkey
on IDs for commenting control forward
slash sounds good yeah yeah okay so I'm
gonna assume that this is going to get
the source of whatever image but I also
need to check maybe there's a table and
there's no image so I'm just gonna add
an extra like if statement to double
check that so if not image then I'm
gonna replace an image with no available
image again and these are just to
prevent some unwieldy errors from
popping up if that does happen okay so
that's my image that's my title the next
step is actually to get the sentences
and this is actually more complicated
because as we saw there's no one
specific tag or whatever so first I'm
gonna get to remember we did this like
thing where we looked around and found
what do the like the text is actually in
so I just identified it and I'm gonna
say the body is souped up find so super
member is the highest level it's a div I
know that and then it has a class of MW
parser output so I actually have no idea
what that means to Wikipedia but as far
as I'm concerned that is where the text
I want it yeah I'm guessing they have
some tool call that's shortened for MW
that maybe is a markdown or something
that compiles to HTML well that's part
I'm in converted to HTML so yeah not
sure but yeah yeah also kind of cool
with inspect element you can see how
certain websites are put together so
that could be something cool to look
into same kind of error checking like if
for some weird reason there is no body
or like no div tag with this class then
I'm gonna say no Wikipedia page
available Thank You Jackalope for the
follow Adam was saying in a web dev
perspective and related to the Photoshop
comment yeah we could
well definitely think about ways we can
integrate them that's a it's a good idea
I don't know if we have too many folks
who have a lot of expertise on it I use
it slightly but I'm not
chef's expert nowhere close yeah so this
is just air checking if there's nobody
I'm gonna say there's nothing and then
one is usually how we signify like an
error has occurred with the program zero
being that like success success yeah
okay so I'm gonna say body text I don't
know how many tags it's gonna be but it
just has an estimate I'm gonna find all
P tags and I'm gonna get five of them so
this is my conservative estimate of the
first sentence will be within five P
tags I'm just going to get all of them
and see what happens
okay so let's see what is inside this so
Adam was saying MW Men's Wearhouse
you're gonna like the way you look I
guarantee it
that's fantastic maybe Oh MediaWiki
MediaWiki okay is the actual I don't
know could be Men's Wearhouse we're
probably not exploit dreams the only
thing that sucks is websites like
Wikipedia are dynamically changing all
the time yes so that goes along with
some challenges of scraping is if they
change it you gotta came you're good
because you're relying on this I mean
even internal inconsistency like we
encountered with the fact that some web
set though images like that alone is
enough of a struggle and then you
compound that with changing formats yeah
yeah yeah well this works for now oh so
okay so I looked at computer science
I've printed a few things this looks
like the first P tag so this is the
beginning of my list and telling me it's
got this class and then all of this
information so it looks like the first
five P tags is probably even too much
but I'm gonna leave it that way for now
and I was just checking to make sure
that body text was working but because
this is kind of like this like I don't
want to give the user this format with
all the P tags so I'm gonna make another
function that is actually gonna parse
this and get the first sentence for me
the true canoes has an interesting
function sorry an interesting question
is there a count function in soup for
counting the number of a specific tag
like H ones whoo I don't know I'm
sure I imagine that's something they
could do this might be a little overkill
but honestly you could just find all of
them and just return the length of that
list yeah that is kind of like
calculating more information than you
mean but it could be a way to do that
yeah hopefully they had to keep it cash
somewhere in the body of the yes super
whatever well if supers grabbing all the
information to begin with it shouldn't
be - I don't know it depends on all
these for their data yeah yeah sure
doing you the primitive way laying on
find all is the consensus right now v1
yeah so here I'm just writing a function
that is going to parse the the massive
piece P tags that I just found and get
the first sentence because that's not
implicit in Wikipedia structure so I'm
gonna say for each tag in a block of
tags where B is my block of tags I'm
gonna say the sentences are getting the
text from all the P tags and then
splitting by period and then for if that
single P tag has a lot of sentences I
want to check that my query so remember
Q is my query and I'm just gonna convert
it to lowercase so it's comparable is in
that sentence that I'm looking at so
maybe it's not the first sentence but I
want the sentence that has the keyword
in it so that's all this statement is
saying and then I would return s I want
that to be my first sentence else I'm
just going to return the first sentence
in this in my P thing
and the parameters here are B and Q what
are those short for yeah give me a
second to I'm gonna boot up away thank
you very much for the photos yes let me
see how I okay so B is going to be body
yeah the body and then Q is gonna be I
think what the query itself okay and so
I'm using Q here to check if the query
is in any of the body text that I've
analyzed and B is just that it's gonna
be able to paint a because I just
generated okay okay so this is my
function for that just kind of walking
through the logic I'm gonna go through
each P tag in the body so I'm saying
like get each chunk and one at a time
I'm going to create this variable
sentences that is going to split that P
tag I'm assuming there are multiple
sentences and that one P tag so this is
going to be a list with a bunch of
sentences and then I'm gonna say within
those sentences give if there is a
sentence where it has the query in it
I'm gonna return that sentence else I'm
just going to return the first sentence
in that chunk so doing a little bit of
checking for word pretty simple so shout
out to Aaron Carvalho in the chat
entropy of life 126 everybody give her a
nice chat oh she is watching the stream
hello hello okay so I've got my body
text and now I'm gonna print everything
out for the user to see so I'm gonna say
print the title is the title variable
then I'm gonna have the description and
this is gonna be my first sentence he
says hi Emily hi and this is gonna be
I'm gonna use the first sentence mention
that I just built you're doing such a
good job says Erin and then these are
just two parameters that I defined the
first being in the body text these are
all the P tags again and then this is
actually the query that I found from the
getstring function all the way up here
it's the original article that the user
was looking for and
I'm just gonna do print the image URL
cool so that should be it
oh this needs I'm gonna run into the
same issue with this being inside I was
gonna mention that earlier than I I got
distracted occasionally okay let's see
if this runs mini wiki I love it by the
way
mini wiki okay let's try computer
science okay so it's saying the title is
computer science of the wiki page here's
the description computer science whoa ah
and it's giving me this Wikipedia link
for the image let's see if that works
oh I think I think it's in black and
it's transparent yeah yeah oh it's right
here actually it's actually pretty
visible on that same not at all visible
at least - can see over here my screen
is dirty
is there another one you want to try
maybe uh well we would try cs50 but they
unfortunately don't have an image on the
page that we can try well I think we did
some error checking for that so yes I'm
turning that there is a page let's try
maybe Harvard University future Harvard
University yeah also medium thank you
very much for the follow so let's try
Harvard so a cs50 it is actually
correctly returning to available right
which is what we wanted we just can't
test for name is against the background
like we would have okay Harvey
okay it looks like my sentence function
is not quite working I don't think
that's not sure if that's the end of the
sentence but that's something it's a
debug later perhaps and we can look at
and what exactly this link is that looks
like the Harvard logo nice maybe like
what happens if there's no Wikipedia
article link to this no like that's
probably not gonna exist and it's
correctly returning the nice a very
robust it looks like the true can you
wanted us to try dog dog okay that's
probably gonna have a cute image so
okay dog oh it looks like okay like a
collage of a nine dogs jpg is the name
of the file okay so that should be
exciting
oh it's interesting they didn't even put
it in the table in that case they just
put all nine dogs together in one image
that's know I like that
why do image links sometimes have slash
slash this slash slash so if we actually
go back to you let's go to Wikipedia and
actually find the dog page that's cool I
didn't realize they have the I never go
to Wikipedia through Google like I
always type WI and press ENTER so I
didn't have a search bar built into then
yeah they do not for a few like any page
that I think has like a built-in search
that's pretty cool yeah okay so here's
the dog page and so we were looking at
the slash slash let's find the image
again see how quick I can do this it's
not here it's in body content okay it's
in here somewhere here here's the table
lower leg due to first table row there's
another is this oh this is actually
before what I want
here's another table room okay here
we're getting to the photo
so this slash slash Boop that's covering
this is showing me the image okay so in
web programming basically all of your
files for your website are within one
domain and the slash slash I think means
start from the like home domain so you
can think of a web site as kind of
organized by files maybe if you're
within it a page within dog or something
and there's more files in there you're
gonna want to look for certain files
within that folder but this slash slash
is just a way to denote that go to the
home directory or I think like with that
means there's like nothing here and then
find this file so it's denoting a
location yeah nice make sense and I'm
was saying she'd generate quads on that
dog image it's a reference to the game
programming stuff idea where you take an
image and splice it up into like
rectangles so you like draw the
individual ones you referenced yesterday
stream Harvard doesn't have that many
students this is true it is quite a
small universe yeah okay and then I
think / - actually means use the same
protocol as the one in the current URL
eg8 okay versus HTTPS yeah I guess oh
shout out to cream Zidane everybody by
the way yeah it's like fYI the entire
office is watching apparently so how
cool that's so fun inflate inflate those
numbers if you want to scrape another
image only one like the graph in the
table from the computer science page how
would you do it if I guess grabbing
another image down the page later on
sorry FIFO asks if you want to scrape
another image only one like the graph in
the table only one image from the
computer science page how would you do
it yeah so remember beautifulsoup has as
I wanted beautiful soup has find and
find all and the difference between
those is one will give you a list of all
of them and one will just give you the
first one and so maybe if you did find
all image it will spit out of the HTML
tags for the various images yeah cool
and apparently the whole office is
watching pererro and rave says hi Emily
: great stream thank you cream excited
to have you on again soon everybody
pressure cream to get on the stream and
Erin pressure and to get on as well
everybody who's kindly shouting him out
be a buggy doodle saying if you can code
under that kind of pressure you're a
winner I've got some shoes with me I
will say they're not monospaced they are
you're using a sans-serif font that
looks like this is that correct yeah
this was on Microsoft Word yeah I'd find
that hard to read but you did it so nice
yeah that was really cool it was
exciting to know I could've vision you
started taking this and making your own
sort of faux Wikipedia page yeah
essentially proxying people's queries
grabbing the actual contents and then
just displaying a very sort of filtered
down version of the Wikipedia pages but
I mean obviously this is the taste of
the kinds of things you can do with
beautiful soup surely you could do as
much as your mind could fathom for sure
it's just a method again for grabbing
information programmatically and then it
really up to you depending on that what
other technologies you know or maybe
interested in learning what you can do
with that information right and
obviously a lot of companies built their
entire infrastructure off of this idea
this is cool well I'm assuming that's
probably it for a beautiful soup today
we actually went over a time a little
bit a little bit our 45 so we had a
great nice long stream today thank you
so much for coming in yeah awesome
I haven't seen beautiful soup in a long
time so it's nice to refresh my mind as
well let's have you again sometime
ideally if you're interested it was
great next week Monday Tuesday will be
doing solitaire again which I did
yesterday I'm gonna also transition us
to the large view here and actually
would you mind sharing this and sending
it to me some and plug it in the chat
yes experiment with it does that be
super awesome Emily's gonna send me that
link and I'm gonna post it here in the
chat paste it here in the chat so
thankfully the cs50 sandbox courtesy of
all of our amazing folks working on it
David creme and others have made it so
that we can share these sort of
sandboxes with other people so we can
let everybody you work on it together
you won't be able to see each other
collaborate in real-time it'll sort of
fork off individual copies of the
workspace but you'll have access to the
code and obviously the pre-installed
libraries that we used today announcing
selector gadget is a Chrome extension
that will give you the name of the
element okay that's cool so sort of like
a Chrome extension version of
beautifulsoup small yeah so maybe that's
a way to do it without coding yeah a
little bit yeah
medium is asking what IDE did you use
the IDE that was usually so this is the
c s-- 5000k mouse which great and magic
we go this is the cs50 sandbox it's
meant for kind of like temporary usage
like maybe you just want to try out a
function or a new technology but don't
really intend to save it locally or
anywhere that's good for that previously
this is I'm told an old version of the
cs50 IDE but basically a development
environment that cs50 has created that
is quite friendly to beginning students
I was using this like I this came up
this is actually Adam it's an ID or it's
not even an ID it's a text editor made
by github I think that was all of them
that I had pulled up yeah for more
information to feel free to
at the sandbox stream that David and I
did this week actually which was super
nice we went to the details on a lot of
that stuff all transitions back to the
larger view here boom and I just got the
sandbox URL so I'm gonna copy that right
on over yeah I'll test it really fast as
well if anybody has any last questions
for the last few minutes before we wrap
up
definitely toss them in the chat we'll
take a look at them it's looking like
it's doing work okay yep it works so I'm
gonna go ahead and toss this right here
into the chat so take a look at that
boom m10 zosh oh the market owns large
nice that's one of the teachers that we
work with their cs50 AP which is super
cool so awesome so again thank you very
much for this awesome stream we'll stay
around for a couple minutes and oh yeah
I was saying next week we have solitaire
Monday and Tuesday so we'll finish up
solitaire which we started yesterday
which is a game like cutting from
scratch and lovin Lua and then we have a
few other things going on I know that
some of the folks won't be able to do a
stream next week but I think on the
Wednesday right we'll be doing cookie
clicker pay for the cookie clicker yeah
anything cookie clicker I have like one
scenarios like what is this we'll be
making one of those from scratch so very
academic you know very academic exercise
and then next Friday stay tuned for a
JavaScript basics lesson which will all
be covering actually I'm doing most of
the streams next week and just realize
that I might be doing every single one
we had a code review set for next week
but unfortunately that'd be postponed so
we doing solitaire Monday and Tuesday
probably because it's a very large
project Wednesday cookie clicker Friday
JavaScript so a little bit of web a
little bit of games lots of stuff this
was cool that I really enjoyed taking a
look at this is kind of separate from a
lot of the domains that we've done so
far it's sort of tangentially related I
mean it's not tangentially but it is
related to web and it's Python but it's
usually we do games and some other stuff
so a lot of fun stuff they make sure if
we catch up on all the chat here
Colton maybe next game stream you could
use unity I've been thinking about this
and I would like to start using unity a
little bit more so let me think a little
bit more about when we can start
transitioning and I think we'll end up
probably doing
that cool cool people are saying alleged
lots of nice words Bella Singh awesome
stream thank you very much Emily and
Colton Ashley thank you very much for
this jam very understandable and
educational that's good that's a plus ma
gravina sending the love which is great
and I know that you feel the struggle
because it's snowing here but miss miss
in California right about now right yeah
I'm really missing California but yeah
i'm i could use a little bit warm
weather myself cool so follow the link
in the chat thank you very much to
everybody who tuned in today for the
stream thanks to Emily again for the
awesome stream on beautiful soup and
Python this was cs50 on twitch and I
will see all of you on Monday have a
great weekend and prepare yourselves for
solitaire part 2 it's gonna be awesome
so see you later oops wrong button