Hello, and welcome to part one of web scraping with Python.
This is a four part introductory tutorial in which you'll use web scraping to build a dataset from a New York Times
article about President Trump. If you'd like to follow along at home, you can download this Jupyter notebook from GitHub and
there's a link to it in the description below.
In
this video,
you'll learn what web scraping is and why it's useful.
As well, I'll explain the three basic facts about
HTML that you need to know in order to get started with web scraping.
So let's start with what is web scraping. On
July 21st,
2017, the New York Times updated an opinion article called Trump's Lies
detailing every public lie the president has told since taking office.
Because this is a newspaper,
the information was of course published as a block of text.
This is a great format for human consumption,
but it can't easily be understood by a computer. In
this tutorial, we'll extract the President's lies from the New York Times article and
store them in a structured dataset.
Now this is a common scenario.
You find a web page that contains data you want to analyze,
but it's not presented in a format that you can easily download and
read into your favorite data analysis tool.
You might imagine manually copying and pasting the data into a spreadsheet,
but in most cases that is way too time-consuming.
A technique called web scraping is a useful way to automate this process.
So what is web scraping?
It's the process of extracting information from
a web page by taking advantage of patterns in the webpage's underlying code.
Let's start looking for these patterns.
Take a second and notice how the article presents its information.
When converting this article into a dataset, you can think of each lie as a
record with four fields.
First is the date of the lie.
Second is the lie itself as a quotation.
Third is the writer's brief explanation of why it was a lie, and
fourth is the URL of the article that substantiates that claim that it was a lie.
Now importantly those four fields have different
formatting which is consistent throughout the article.
The date is bold red text,
the lie is regular text,
the explanation is gray italics text, and
the URL is linked from that gray italics text.
So why does the formatting matter?
Because it's very likely that the code
underlying the webpage
tags those fields differently and we can take advantage of that pattern when scraping the page.
So let's take a look at the source code for this page known as HTML.
To view the HTML code that generates a web page, you right click on it and
select View Page Source in Chrome or Firefox,
View Source in Internet Explorer, or Show Page Source in Safari. Now
if that option doesn't appear in Safari,
just open Safari preferences, select the Advanced tab, and check "Show Develop menu in menu bar."
Again, that's for Safari users only.
So, notice the first few lines you'll see when you're viewing the source of the New York times article.
Then, let's locate the first lie by searching the HTML for the text Iraq.
So Ctrl+F and then "iraq".
Thankfully you only have to understand three basic facts about HTML in
order to get started with web scraping.
Fact one is that HTML consists of tags.
You'll see that the HTML contains the article text
along with tags
specified using angle brackets
that mark up the text. HTML actually stands for Hypertext Markup Language.
For example, one tag is strong,
which means use bold formatting. There's a strong tag before January 21st and
a strong tag after
January 21st.
The first is an opening tag and
the second is a closing tag
denoted by this
forward slash
which indicates to the web browser where to start and where to stop
applying the formatting. In other words, this tag
tells the web browser
to make the text January 21st bold.
Now don't worry about this nbsp. We'll deal with that later.
Okay, fact two:
HTML tags can have attributes which are specified in the opening tag. For example,
span class equals short-desc
indicates that this particular span tag
has a class attribute
with a value of short-desc.
Now for the purpose of web scraping you don't actually need to understand the meaning of
span class or short-desc.
Instead you just need to recognize that tags can have attributes and that they are
specified in this particular way.
So that's fact 2.
Fact 3 is
that tags can be
nested.
Let's pretend my HTML code said hello
strong em Data School em students strong.
Okay?
The text Data School students
would all be bold because all of that text is between the opening strong tag and the closing
strong tag.
The text Data School
would also be in italics because the em tag means use italics.
The text Hello
would not be bold or italics because it's not within either the strong or
em tags.
Thus it would appear as follows:
Hello
Data School (bold and italics) and then students (just in bold).
The central point to take away from this example is that
tags
mark up text from wherever they open to wherever they close
regardless of whether they are nested within other tags.
Okay, if you've got that, you now know enough about HTML in order to start web scraping.
In the next video, we'll read the New York Times article into Python,
parse its HTML using the Beautiful Soup
library, and then start building our dataset by taking advantage of the patterns we noticed in the article
formatting.
If you have a question or a tip, let me know in the comments section below.
Please click subscribe if you like this video.
Thank you so much for joining me and I hope to see you again soon!