Next we will look at the problem of web spam.
So the problem actually became very,
important with the rise of the web search engines.
So let me first define what do I mean web spam, right?
So spamming, it's really any deliberate activity to boost the score
of a webpage or the webpage position in the search engine results page.
All right?
And this means that basically what, what,
the goal is to kind of boost the value of a given page.
so,spam, in some sense, are webpages that are the result of spamming.
And actually, there is a whole industry of people, in some sense,
that are trying to do this.
And this is what is called a search engine optimization, right?
And there is a kind of grey line, and these people might disagree.
But basically, the idea is, how do we boost an importance of a,
of a given page on the web?
And make, make sure it appears, let's say, on the first page of Google or
Bing results.
And it has been estimated that around 10 to 15% of webpages on
the web are actually spam.
So how did spam web spam really work?
So early search engines operated the following way.
They would go and crawl the web.
So basically they would go and collect the web pages.
They would then index the web pages by the words they contained.
And then whenever somebody searched, they would basically look at
the given map page, see how much does it mention the words in the search query, and
then it would rank the search results is
based on the number of times they mention a given, search criteria.
So, what this means is that basically the idea for the initial ranking was
to attempt toward the pages by matching a a search query by importance.
Which means that basically the search engines would,
would consider the number of times query words appeared on the webpage, and
maybe also the prominence of the word position,
whether that word was in the title or in the header of a webpage.
So now that also is very, allows us to nicely, nicely spam and exploit this idea.
So imagine the following.
Imagine that when peoples began using the search engines for, on the web.
Those that, with the commercial interests.
Maybe people that are wanting to sell something.
They want to exploit the search engines and
make sure that people go and visit their own pages.
Right? So, the idea is kind of we want our
page to appear as high, as high on the web search ranking as possible,
regardless of whether that is, that page is really irrelevant to our given topic or
not, because driving traffic to our website is good.
So lets look at the example where we have our t-shirt reseller who wants to
appear high on the, whenever people search about movies.
Maybe these t-shirt reseller reselling movie related t-shirts, so he wants to
be as high as possible whenever somebody searches for the word movies.
Right. And the technique to achieving this,
boost in the relevance or
importance of our web page, this is really what we mean by spam.
So here is how, how the first, spammers did this.
This is called term spam.
So the idea is the following.
Right?
The idea is that for our t-shirt seller, he creates the web page, and
then, wants to appear this web page to really seem to be about movies.
So what the web seller can do, is he can take the word movie, and
insert the word mo, movie, 1,000 times in the webpage.
And usually how this was done in the early days on the web, is that you,
you would have a webpage where on the top was the legitimate text, and
at the bottom was a huge list of words and the ca,
the text color of these words was the same as the color of the web background.
So kind of they wouldn't bother the user who,
who comes to the web page, but the web search engine would see these words and
would think that this particular web page is all about movies, all right?
And now of course when somebody will see and run a query on,
on movies, this web page would appear to be all about movies, and
the person would go and get it ranked higher.
So, this and similar techniques to this one, are called term spam.
Let's look at how Google came up with the solution to combat term spam.
So the basic idea, and the basic intuition here is that,
rather than believing what the page says about itself,
let's believe other people, what they say about the page.
So in our context, this means that we want to look at the anchor text.
So this is the blue text underlying the hyperlink.
What does that text say about the page that it is linked to?
So the, in some sense the idea is that we are able to find web pages and, and
surface them out even if even for queries for
the words of it that the web page itself doesn't even mention, but other web
pages on the web may mention that word when referring to the target page.
And of course, now we already know how we can use PageRank, so we can think of
PageRank as a tool to measure the importance of web pages on the web through
this idea of the anchor text, and the question is why does this method works?
All right, in our hypothetical t-shirt seller example, basically the,
we can combat him in a sense that now, say, if it was very easy for
the t-shirt to say, to put the word movies on the web page many times, and
say hey, I'm really about movies and not about t-shirts.
But from the t-shirt seller's perspective,
it is very hard to convince other pages on the web to link now to our teller's,
t-shirt seller's webpage, and say that this webpage is about movies.
Of course, and
this is now where we are basically coming to the notion of web spam.
Where the idea is that what a web seller can do now, they can go and
create 1,000 web pages, that each link to his, t-shirt seller web page and
each of these thousand web pages use the word movie in, in the anchor text.
And this way, for example, now we are having the notion of web spam where
the idea is that these web pages have are creating false
links to our t-shirt seller web page, in this way boost the importance of the page.
Of course, one thing is now that, for example, these pages have no so they will
get very little web page page rank score, and this page may still be linked low.
So the, the, the web shirt, t-shirt seller person, it's very hard for
them, for them, for example to beat a truly movie related webpage like IMDB.
Of course even this scheme can be beaten, for example,
if there are big coordinated efforts where people put a given anchor text and
link to a given target page, then what can happen is that this given target page can
be ranked very high, even though it doesn't mention a word.
Here is an example about a coordinated effort,
about a given query, in about a given target page.