Hi Everyone! so in the past few days, I'm sure you guys noticed
I had a chance to go over many many viewer comments
and to comment them back
now what I noticed on my end is that
a lot of people have misconceptions about web scraping
and I really don't blame you guys because there's a lot of
conflicting information online
some articles are claiming that it's super illegal
some articles are claiming that it's perfectly legal
so I thought I might as well film a video
to share my experience with web scraping
now the topics we will talk about in this video are
A) is web scraping legal or not
B) are the terms of use or community guidelines
of platforms allow web scraping
and 3) we will talk about copyrights and fair use
and before we move on - I have to mention
that I am not a lawyer, I am not a legal consultant
I am not supposed to give you legal advice
everything I'm gonna say in this video - please double check it on your end
read some articles, see some examples
don't just rely on my words
even though I'm a very reliable person - but it up to you to judge
now to determine whether web scraping is legal or not
we will actually use some basic logic
and my first argument is: there is absolutely nothing
you can do with Selenium or Mechanical Soup or Beautiful Soup
that you cannot do manually with your mouse and keyboard
for example: if I wanted to connect with many many people on LinkedIn
I could have just clicked on the "connect" button
page after page and connect with as many people as I wanted
there is absolutely no difference between the results of manual connections
and the results of connections via bots
we are simply using our programing skills
to make our life slightly better
so my philosophy is: as long as you can do it with your mouse and keyboard
there is no reason why you can't do it with automation
now my second argument would be
that many professional developers are using web scraping on a daily basis
they work in legitimate companies
some of them are doing marketing, some of them are doing human resources
some of them are doing cyber security
automation and web scraping is part of their daily routine
how would you test software or web applications if you cannot use automation?
and why would libraries like Selenium, Mechanical Soup and Beautiful Soup
exist in the first place if there's something wrong about that practice
now let's talk about Terms of Use of certain platforms
so if we take Instagram, for example
they strictly forbid the use of crawlers, bots or anything automated
that is not related to search engines
and yet! many people are still scraping Instagram!
so the worst thing that Instagram or any other social media platform can do to you
if you're violating their community laws
is to delete your account!
and for some of us - deleting their social media account would be devastating
for others - we don't really mind
there's nothing that prevents us from opening a brand new account
which can be a fake account
it doesn't even have to have our name
and to web scrape from there
so what people usually do is they have a personal account, or a company account
that is being regulated by humans, of course
and on top of that, they have almost an army of bots
that consists of "dummy" accounts
and when I say "dummy accounts" I don't mean "stupid accounts"
I mean "fake accounts"
that are only used for advertising purposes
for example: if you post a brand new photo on Instagram
and you add the #fashion or #style
you will most likely get a comment on that photo
asking if you would like to collab
with some kind of a jewelry company or some kind of a clothing company
now the comment itself is generated by a bot
that usually doesn't even have any followers
but in the body of the message -
the bot will include a link to the official account of the brand
that has many many different followers
and seems a 100% legit!
and he will ask you to directly message the official company
and not the bot
but the truth is: they don't really care about your 300 followers
they don't care about your influence
maybe, once in a blue moon, somebody grows and gets enough followers
based on these partnerships
but usually - the only customers that these brands have
are those influencers that directly messaged them as a respond to the bot
so yes, they offer you discount on their products
and yes, sometimes they tell you - "we will send you the products for free"
"you just need to pay for shipping"
but guess what?
they most likely don't have any other customers
so it's a brilliant way to market your fashion company,
at least in my opinion
and that's why I believe that Instagram came up with these regulations
in the first place - they just don't want people to be tricked!
and with all due respect to the terms of use
and to the community regulations of each platform
these are not laws!
so can you get in legal trouble for automating a social media platform
that doesn't want to be automated?
NO! but they can remove your account
or they can suspend it for a week or two weeks
that's pretty much all
mostly likely - they will not even know that you web scraped anything
because nobody cares about the 30 photos of cats
you downloaded to your computer
that's not the purpose of these regulations
what CAN get you in legal trouble
is copyrights! and this is where I need you guys to pay extra attention
and again, I am not a lawyer, I am not a copyrighter
everything I know about copyrights and fair use
I learned on the internet
yes, I have read many different articles
but I still want you guys to do your own research
so whenever I talk about copyrights
what I actually mean is - what are you planning to do
with the information you web scraped
so let's say - you just downloaded 40 images of cats to your computer
did you violate any copyrights laws?
no. not yet!
but if you're planning to distribute these photos
this is when copyrights come to play
so (among) the only few things you can do with the information you web scraped
is using it for educational purposes
and for non-profit purposes
because whenever you're trying to make profit
out of something that you didn't create
you're not the original owner of this content
there's really no difference between this and stealing
the only difference is you're stealing intellectual property
and not physical property
but if the only thing you're doing with this information
is saving it on your computer
or using it for educational / non-profit purposes
you are not violating anything!
you're actually protected under the Fair Use clause
of copyrights
people who claim that if you copy anything from the internet
if you download anything you saw online
you are violating somebody's copyrights
these folks are not familiar with the Fair Use clause
and in fact, if this clause didn't exist
most likely the internet would not exist
because the whole purpose of the internet
is to share information one with another
so before you let people freak you out
about copyrights - please read about fair use!
now thank you so much for watching and I will see you guys very soon
in my next video - where we finally talk about Artificial Intelligence
so stay tuned!