SUZ HINTON: A lot of confusion revolves around SEO
because no one understands how the Googlebot actually works.
[MUSIC PLAYING]
MARTIN SPLITT: Hello and welcome to another episode
of "SEO Mythbusting."
With me today is Suz Hinton from Microsoft.
Suz, what do you do at work, and what is your experience
with front end SEO?
SUZ HINTON: Yeah, so right now, I'm
doing less front end these days.
I focus more on IoT.
MARTIN SPLITT: So in the time you were a front end
developer--
SUZ HINTON: Yeah, I was a front end developer for, I think,
12 or 13 years.
And so I got to work on lots of different contexts of front end
development, different web sites, things like that.
MARTIN SPLITT: Cool.
SUZ HINTON: Today, I wanted to just
address a bunch of stuff about Googlebot specifically,
and nerd out about Googlebot, because that
was the side of things that I was the most confused about
at the time.
MARTIN SPLITT: So Googlebot is basically a program
that we run that does three things.
The first thing is it crawls, then it indexes,
and then last, but not least, there's
another thing that is not really Googlebot anymore.
That is the ranking bit.
So we have to basically grab the content from the internet,
and then we have to figure out what is this content about?
What is the stuff that we can put out to users
looking for these things?
And then last, but not least, is which
of the many things that we picked for the index
is the best thing for this particular query
in this particular time?
SUZ HINTON: Got it, yeah.
MARTIN SPLITT: But the ranking bit, the last bit,
where we move things around-- that is informed by Googlebot,
but it's not part of Googlebot.
SUZ HINTON: Is that because there's
this bit in the middle, the indexing?
The Googlebot is responsible for the indexing
and making sure that content is useful for the ranking engine
to--
MARTIN SPLITT: Absolutely, absolutely.
You can imagine, someone has to-- in the library,
someone has to figure out what the books are about
and get the index of the bits in a catalog, the catalog
being our index, really.
And then someone else is using that index
to make informed decisions and going, here,
this book is what you're looking for.
SUZ HINTON: I'm really glad you used
that analogy because I worked in a library for four years.
MARTIN SPLITT: So you know much better than I how that works.
SUZ HINTON: And I was that person.
People would be like, I want Italian cookbooks,
and I'm like, well, it's 641.5495.
And you would just give it to them.
MARTIN SPLITT: If I would come to you, as a librarian,
and ask a very specific question,
like so what is the best book on making apple pies really quick,
would you be able to figure out, from the index--
you probably have lots of cookbooks.
SUZ HINTON: We did, yeah.
We had a lot.
But given that I also put lots of books back on the shelf,
I knew which ones were popular.
I've no idea if we can link this back to Googlebot.
MARTIN SPLITT: That does.
Yeah, it's pretty much-- so you have the index that probably
doesn't really change that much, unless you add new books to it.
SUZ HINTON: New editions.
MARTIN SPLITT: Exactly, yeah.
So you have this index, which Googlebot provides you with.
But then we have the second--
the librarian second part that basically is,
based on how the interactions with the index work,
figure out which books to recommend
to someone asking for it.
So that's pretty much the exact same thing.
Someone figures out what goes into the catalog,
and then someone uses it.
SUZ HINTON: I love this.
This makes total sense to me.
MARTIN SPLITT: But I guess that's still not necessarily
all the answers you need.
SUZ HINTON: Yeah, I just want to know, what does it actually do?
How often does it crawl sites?
What does it do when it gets there?
What does it-- how is it generally behaving like?
Does it behave like a web browser?
MARTIN SPLITT: That's a really good question.
Generally speaking, it behaves a little bit like a browser--
at least, part of it does.
So the very first step, the crawling bit,
is pretty much a browser coming to your page,
either because we found a link somewhere,
or you submitted a site map, or there's
something else that basically fit that into our systems.
You can use Search Console to give us a hint and ask
for re-indexing, and that triggers a crawl before--
SUZ HINTON: I've done that before.
MARTIN SPLITT: Oh, very good.
SUZ HINTON: We asked for it to be done.
MARTIN SPLITT: And that is perfectly fine,
but the problem then, obviously, is how often do you
crawl things, and how much do you have to crawl,
and how much can the server bear.
If you're on the backend side, you
know that you have a bunch of load,
and that might not be always the same thing.
If it's like a Black Friday, then the load
is probably higher than on any other day.
So what Googlebot does is it tries to figure out,
from what we have in the index already,
is that something that looks like we
need to check it more often?
Does that probably change?
Is it like a newspaper or something?
SUZ HINTON: Got it, yeah.
MARTIN SPLITT: Or is that something
like a retail site that does have offerings that
change every couple of weeks?
Or even do not change at all because this is actually
the site of a museum that changes very rarely?
For the exhibitions maybe, but a few bits and pieces
don't change that much.
So we try to like segregate our index data into something
that we call daily or fresh, and that gets
called relatively frequently.
And then it becomes less and less frequent as we discover,
and if it's something that is super spammy or super broken,
we might not crawl it as often.
Or if you specifically tell us, do not index this,
do not put this in the index, this
is something that I don't want to show up
in the search results, and we don't come back
every day and check.
So you might want to use the re-index feature
if that changes.
You might have a page that you go, no, this shouldn't be here,
and then once it has to be there,
you want to make sure that we are coming back and indexing
again.
So that's the browser bit.
That's the crawler part, but then a whole slew of stuff
happens in between that happening,
us fetching the content from your server,
and the index having the data that is then
being served and ranked.
So the first thing is we have to make sure
that we discover if you have any other resources on your page.
The crawling cycle is very important.
So what we do is, the moment we have some HTML from you,
we check if we have any links in there,
or images for that matter, or video--
something that we want to crawl as well,
and that feeds right back into the crawling mechanism.
Now, if you have a gigantic retail site,
let's say, just hypothetically speaking,
we can't just crawl all the pages at once,
both for our resource constraints,
but also we don't want to overwhelm your service.
So we basically try to figure out
how much strain we can put on your service
and how much resources we've got available as well,
and that's called the crawl budget, oftentimes.
But it's pretty tricky to determine, so one thing
that we do is we crawl a little bit,
and then basically ramp it up.
And when we start seeing errors, we
ramp it down a little bit more.
So oops, sorry, for that, we are not-- oh, ugh.
So whenever your service serves us 500 errors,
there are certain tools in Search Console that
allow you to say, hey, can you maybe chill out a little bit.
But generally, we don't try to get all of it at once
and then ramp down.
We are trying to carefully ramp up, ramp down again, ramp up
again, ramp down again, so it fluctuates a little bit.
SUZ HINTON: There's a lot more detail in there
than I was even expecting.
I didn't even know that--
I guess I never considered that a Googlebot crawling
event could put strain on somebody's website.
That sounds like it's a lot more common than I even
thought it would be.
MARTIN SPLITT: It does happen, especially
if we discover, say, a page that has lots
of links to subpages pages.
Then all of these go into the crawling queue,
and then you might--
let's say you have 30 different categories of stuff,
and each of these have a few thousand products and then
a few thousand pages of products.
So we might go, oh, cool, crawl, crawl, crawl, crawl, crawl,
crawl, crawl, and then we might crawl a few hundred
thousand pages.
And if we don't spread that out a little bit--
so it's a weird balance.
On one hand, if you add a new product,
you want that to be surfaced and searched as quickly
as possible.
On the other hand, you don't want
us to take all the bandwidth that your server offers.
I mean, cloud computing makes that a little less scary,
I guess, but I remember the days--
I'm not sure if you remember the days where
you had to call someone, and they ask you
to send a form or fax a form.
And then two weeks later, you get the confirmation letter
that your server has been started.
SUZ HINTON: Yes, I remember the days
when we would have to call, and then we would basically
pay $200 to have a human go down the aisles
and push the physical reset button on the server, so yeah.
MARTIN SPLITT: Those times were a lot trickier, yeah.
And then imagine you basically renting five servers somewhere
in a data center, and that taking a week,
and then we come and scoop up all your bandwidth.
And you're like, great, we're offline today
because Google has its crawl day.
That's not what we want to have.
SUZ HINTON: Yeah, these days, it's
more like a happy news kind of moment, when you get hit.
MARTIN SPLITT: Exactly.
SUZ HINTON: So I feel like you're
much more considerate than--
MARTIN SPLITT: Yeah, we try to not overwhelm anyone,
and we respect the robots.txt.
So that works within the crawl step as well.
And once we have the content, we can't
put strain on your infrastructure
anymore, so that's fantastic.
But modern web apps being mostly JavaScript driven,
we then put that in a queue, and then
once we have the resources to render it,
we actually use another headless browser kind of thing.
We call that the Web Rendering Service.
Then there's other crawlers as well
that might not have the capacity or the need to run JavaScript.
This is like social media bots, for instance.
They come and look for metadata.
If that meta tag is coming in with JavaScript,
you usually have a bad time, and they're just like, sorry.
SUZ HINTON: Yeah, so that's always been a big mess,
and I remember when single page applications, or SPAs,
really came into vogue.
A lot of people were really concerned.
There's a lot of FUD around.
Well, if crawlers in general don't execute JavaScript,
then they're going to see a blank page,
and how do you get around that?
So contextually, within Googlebot,
it sounds like Googlebot executes JavaScript--
MARTIN SPLITT: They do.
SUZ HINTON: Even if it does do it at a later point.
MARTIN SPLITT: Yes, correct.
SUZ HINTON: So that's good?
MARTIN SPLITT: That's good.
SUZ HINTON: But is there anything
that people need to be aware of beyond just,
oh, well, it'll just run it, and then
it'll see exactly the same thing as a human with a phone
or a desktop would see?
MARTIN SPLITT: There's a bunch of things
that you need to be aware of.
So the most important thing is, again, as you said,
it's deferred.
It happens at a later point.
So if you want us to crawl your stuff as quickly as possible,
that also means we have to wait to find these links
that JavaScript injects.
Basically, we crawl, we have to wait until JavaScript
is executed, then we get the rendered HTML,
and then we find the links.
So the nice little short loop that
finds these links relatively quickly right after crawling
will not work.
So we will only see the links after we render it,
and this rendering can take a while because the web is
surprisingly big.
SUZ HINTON: Yeah, just a little bit.
MARTIN SPLITT: There's 130 trillion docs in 2016, so--
SUZ HINTON: So there's way more now.
MARTIN SPLITT: There's way more now.
There's way more than that.
SUZ HINTON: So robots.txt is very
effective at being able to tell bots how to do a certain thing.
But in this scenario, how do you tell
that it's Googlebot visiting your site as opposed
to other things?
MARTIN SPLITT: So as we are basically
using a browser in two steps-- one is the crawling,
and one is the actual rendering--
both of these moments, we do give you the user agent header.
But basically, there's the string--
literally the string Googlebot in it.
SUZ HINTON: That's so straightforward.
MARTIN SPLITT: Yes, and you can actually
use that to help with your SPA performance as well.
So as you can detect on the server side,
oh, this is Googlebot user agent requesting,
you might consider sending us a prerendered static HTML
version, and you can do the same thing for the others.
All the other search engines and social media bots
have a specific string saying that they are a robot.
So you can then basically go, oh, in that case,
I'm not giving you the real deal, the single page app.
I'm giving you this HTML that we prerendered for you.
It's called dynamic rendering.
We have docs on that as well.
SUZ HINTON: The one thing that still doesn't quite
make sense to me is does the Googlebot
have different contexts?
Does it sometimes pretend that it's--
I think of it as this little mythical creature
that's pretending to do certain things.
So does it pretend to be on a mobile, and then desktop?
Are the different, I guess, user agents,
even though it still says Googlebot?
And can you differentiate between them?
MARTIN SPLITT: You're asking great questions, because yes,
we have different user agents.
So I'm not sure if you heard about mobile first indexing
being rolled out and happening.
SUZ HINTON: I've heard that it's going to affect
how you're ranked potentially.
MARTIN SPLITT: That as well.
SUZ HINTON: I don't know if that's a rumor or not, yeah.
MARTIN SPLITT: Ah, that's two different things
that get conflated so often.
So mobile first indexing is about us discovering
your content using a mobile user agent and a mobile viewport.
So we are using mobile user agents,
and the user agent strings says so.
It says something about Android in the name,
and then you're like, aha, so this is the mobile Googlebot.
We have documentation on that.
There's literally a Help Center article
that lists all these things.
So we try to index mobile content
to make sure that we have something
nice to server for people who are on mobile,
but we're not pretending random user agents or anything.
We stick to the user agent strings
that we have documented as well, and that's
mobile first indexing, where we try
to get your mobile content into the index rather
than the desktop content.
Then there's mobile readiness, or mobile friendliness.
If your page is mobile friendly, it
makes sure that everything is within viewport,
and you have large enough tap targets and all
these lovely things, and that just is a quality indicator.
We call these signals.
We have over 200 of them.
SUZ HINTON: That's a lot.
MARTIN SPLITT: Right?
So Googlebot collects all these signals
and then stuff them, as metadata, into the index.
And then when we rank, we're like, so this user's on mobile,
so maybe this thing that has a really good mobile friendliness
signal attached to it might be a better one than the thing
where they have to pinch zoom all the way out
to be able to read anything, and then can't actually
deal with the different links because they're
too close to each other.
So that's one of the many--
it's not the signal.
It's one of the many signals.
It's one of the over 200 signals to deal with.
SUZ HINTON: I had no idea there were 200.
That's making me--
I know that you're not allowed to share what they all
are because there has to be a certain mystique around it,
because of, I guess, a lot of SEO abuse in the past.
MARTIN SPLITT: Yeah, yeah, unfortunately, that
is a game that is still being played,
and people are doing weird stuff to try to game us.
And the interesting thing with this is, with the 200 signals,
it's really hard to say which one
gets you moving in the ranks.
SUZ HINTON: The weights of each signal because--
MARTIN SPLITT: And they keep moving, and they keep changing.
I love when people are like, no, let's do this, and then, look,
my rank changes.
Yeah, for this one query, but you
lost on all the other queries because you did really
weird and funky stuff for that.
So just build good content for the users,
and then you'll be fine.
SUZ HINTON: I feel like that--
it feels like less effort as well,
than constantly trying to--
MARTIN SPLITT: Yeah, but it's not an easy answer.
You pay me to make you more successful on search engines,
and I come to you and say, so who are your users,
and what do they need, and how could you
express that so that they know that it's what they need?
That's a hard one because that means I basically
bring the ball back to you, and now, you
have to think about stuff and figure it out, strategically.
Whereas if I'm like, I'm just going to get
you links or do some funky tricks here,
and then you'll be ranking number one.
That's an easier answer.
It's the wrong answer, but it's the easier answer.
So people are like, links are the most important metric ever,
and I'm like, no.
We have over 200, and it's important,
but it's not that important.
And chill out, everybody.
But this still happens.
SUZ HINTON: I'm so glad it's better now.
I feel, actually, more at peace in general with SEO, as well,
after speaking to you today.
MARTIN SPLITT: Ah, so good.
Suz, thank you so much for being with me
here, and has been a great pleasure.
SUZ HINTON: Yeah, thanks for answering
all of my weird and wonderful questions about the Googlebot.
MARTIN SPLITT: Perfect questions.
Perfect opportunity.
Did we bust some myths?
SUZ HINTON: I feel like we did.
MARTIN SPLITT: Fantastic.
I think that's worth a high five.
SUZ HINTON: Awesome.
Thanks.
MARTIN SPLITT: Thanks.
Join us again for the next episode of "SEO Mythbusting,"
where Jamie Alberico and I will discuss
if JavaScript and SEO can be friends and how to get there.