In this video we're going to be discussing web scraping. What is web scraping,
and how is it being used online. If you enjoy this kind of content, make sure to hit the like button
and subscribe to this channel and turn on notifications so that you don't miss a future
episode. With that out of the way let's get into this video. Today we're talking about web
scraping. What is web scraping? You may have heard this term thrown around by programmers.
Web scripting is the ability to crawl a website or use a bot to gather data from a website
and use that data to do something on a different server, or on your own website. You're going to
take that data from a third-party website use it for your own means. So when it comes to why people
would use a web scraper, there are literally dozens if not hundreds of different reasons
why people use web scrapers. Let's look at some of the common reasons. People will build things
like a comparison website for example. So that will pull data from multiple different sources,
and allows the users of that website to compare something like prices or
details about different products for example. That's one use case where you want to pull the
data off multiple sources. There are other reasons why people use web scraping. Some of
them are very harmless reasons, and there are also reasons why people do it that are a little bit
more concerning. Some people use web scraping to gather people's private data for example.
Email addresses or phone numbers, things like that. So there are bad reasons why people do it
but there are also good reasons or harmless reasons why people do web scraping. Another
example of a case where you may want to do web scraping is something like a search engine so
for example google has its own crawlers. Web crawlers so in that case they are literally
just going to your website downloading the data caching it and indexing information on that page
so maybe they're counting the keywords and what those keywords are on the page, and indexing it
in their search results. So literally these big search engines are doing that at mass scale,
doing that across millions of websites. Crawling them, downloading data and processing that data.
That's one real world use case where web scraping or crawling is basically happening all the time,
every day. There are other use cases where there are business needs for doing web scraping.
Another example is if you wanted to gather information from a social media platform. So
maybe you're doing something like brand awareness, so you have a big brand and you want to know what
people are saying about their products or their services. You can do something like
sentiment analysis on that data so whether people are happy with your products or dislike them
so you could do something like that, where you're scraping data off social networks. In that case
you would be processing the data and you would be doing some kind of sentiment analysis on that data
to understand what the meaning behind that data is. That's a different example of how you might
do web scraping. Quite often the use for web scripting is also to analyze your competitors.
Many businesses do this. They want to understand when their competitors are changing prices
or when they're offering new services for example. So they use web scraping as a means of monitoring
their competitors and seeing what they're doing differently. What they're changing, so that is
kind of a different use case again. In general, I would say that web scraping is not that difficult.
It does vary in terms of difficulty depending on which website you're trying to scrape.
For example if you are trying to get data that is behind a login screen. Maybe you're trying to
scrape a large website and they require you to sign into that account. Well then you wouldn't
be able to scrape the data that is behind that login. In that case you would need to do some kind
of authentication before scraping. That gets a little bit more complicated but for public pages,
web scraping is not that difficult generally. And it really just comes down to putting the
right practices in place so that you are not causing problems on their website. When you are
scraping web pages that are public then it's much easier because there are no complications like
sign ins or logins to access that data, it's public information. You can scrape it but you
should bear in mind that you are also dealing with someone's live web server, so you don't want to
crash their server. You don't want to be sending a lot of traffic at their website at one time,
so there are measures you should consider to do or to put in place to prevent causing a problem
on their website. Normally i would recommend doing some kind of delay between requests so putting a
one second or two second delay between requests to prevent you from overloading their website
if they have hundreds of pages. There are things you can do to make the process easier but this
is just a quick summary video to talk about web scraping and the reasons you might want to use it
as a developer. That's about it for this video, hopefully you guys have found this helpful.
Make sure to like and subscribe and drop me any questions in the comments below,
and as always thank you guys for watching and i will see you all in the next video.
you