[MUSIC PLAYING]
MARTIN SPLITT: Hello, and welcome, everybody,
to another episode of "SEO Fairy Tales."
With me today is Dave Smart, technical SEO and dev
at Tame the Bots.
And I just found out you invented purple ceramics
for use in jet engines?
DAVE SMART: I did, indeed, yeah.
MARTIN SPLITT: How did that happen?
DAVE SMART: That was my first job out of school and out
of uni.
I did interior science in an aerospace lab
and got to work on that.
It was great fun.
MARTIN SPLITT: What?
Like-- first-- my first job was a lot more boring than that.
But that is amazing.
So you kind of did rocket science.
DAVE SMART: Kind of.
MARTIN SPLITT: Ish.
DAVE SMART: Ish.
MARTIN SPLITT: I mean, it's jet engines.
It's cool too.
DAVE SMART: Rocket science lite.
MARTIN SPLITT: Fair.
We're not going to talk about rocket science lite,
unfortunately, but something that is somewhat
looking like rocket science to a lot of people,
which is technical SEO.
[HARP PLAYING]
You brought a technical SEO challenge to me
that you experienced.
What is that?
DAVE SMART: It was to do with service workers.
MARTIN SPLITT: Oh.
Yeah?
No, OK, yeah.
Mm-hmm.
DAVE SMART: Twice it's actually happened.
Two people have approached me now
with sites that just didn't quite work.
One, it just didn't work at all, and they
were relying very heavily on the service worker.
So they were just ending up with blank pages and no indexing.
And the other was related but not maybe quite
so serious in some ways.
But they only end up with partial indexing.
MARTIN SPLITT: But that to me is confusing
because for those of you who don't service workers,
the idea is it's kind of an add-on thing.
It adds to the functionality of a site
but does not necessarily--
or not per se have such an impact because what
it should be used for--
I'm going to be treading carefully here--
is that it should basically just be
a layer between your site and the network.
And that allows offline capabilities and push
notifications and background sync, and stuff.
So what did they do?
[HARP PLAYING]
DAVE SMART: When the first case--
when you have a service worker on a site,
you need to register it.
MARTIN SPLITT: Yes, correct.
DAVE SMART: So the browser will come along
and find the reference to the service worker, register it.
MARTIN SPLITT: Try to register it.
DAVE SMART: Try to register it, even.
However, what they were doing is waiting
for that to happen before they were doing all
the other client-side rendering things,
so before they went and fetched all the data
and put everything to do, which kind of worked for a user.
Could be a little slow because you
have to wait for that service worker to register.
However, for something like Googlebot,
it doesn't register something.
It also [INAUDIBLE] create [INAUDIBLE]..
MARTIN SPLITT: That's actually not even a bug or anything
because the idea is, again, it's an optional feature that you
might opt out of, or browsers might not support it.
So you are requesting it to be registered,
and it might fail or never register.
And we actually document that behavior
because a service worker might change the behavior
for a returning visitor versus a first-time visitor.
And we assume that people on Search results
are first-time visitors.
[HARP PLAYING]
How did you figure that one out?
Because I think that debugging that is not super easy, is it?
DAVE SMART: Yeah, it's not always very easy.
And basically, a lot of what you're doing
is looking for stuff that isn't there that should be.
So the first thing you can do with them, with Search Console,
we could look and see there's very little actually indexed.
The very-- and lots of things like software FOs.
So this looks like an error page.
The few pages that were indexed were basically just the shell.
So when a site is client-side rendered,
they send just a minimal shell.
MARTIN SPLITT: It's a placeholder, basically,
until everything gets fetched.
DAVE SMART: That's right.
And, basically, that's as far as everything got.
So then it's on to the live testing tools.
So things like URLinspector does a lot of tests.
There's Rich Results test.
There's the Mobile-Friendly test.
If you run it through them, you can see much the same result.
It's empty.
However, what you do get is, under the More Info,
a little tab you'll find on the side,
you can see the resources that are loaded in
or haven't loaded in.
MARTIN SPLITT: True, you can see the requests being made.
DAVE SMART: That's it.
So you can see if they've failed,
if they've been successful, or if they're
just not been tried at all.
MARTIN SPLITT: I was about to say, let me guess,
you probably saw them not even being tried.
DAVE SMART: Exactly.
Yeah.
MARTIN SPLITT: Oh, and then the conclusion would be,
oh, JavaScript and Google Search does not work.
DAVE SMART: Yeah, doesn't work.
Yeah.
What there also is is, there is a JavaScript console as well.
So that can give some useful feedback.
But it depends a little bit on how
the site is made because it depends
on what they're logging out.
MARTIN SPLITT: Yeah, exactly.
If you have no error logging, nothing ends up in the console.
DAVE SMART: So at that point, it's time.
Go and make friends with your devs.
[HARP PLAYING]
Go and ask them.
Basically, that should be your first step
after you discover that there is a problem.
Go to your devs.
MARTIN SPLITT: But that's a tricky one
because if you don't know what's going on,
what are you going to talk about with them?
DAVE SMART: Yeah, well, that's the thing.
You don't necessarily know how they built the site.
They know how they built the site.
You know there's a problem.
So go and talk to them, and you start from the first step.
You make little breadcrumbs on the trail through the path.
And you talk through the process of how
does data come from your server and end up on the screen.
So each little step along the way, and then--
MARTIN SPLITT: Yes, and then you can buddy up and become
like a detective tag team.
That's cool.
DAVE SMART: And get them to log each little stage
in a JavaScript console log, something like that.
MARTIN SPLITT: And you can see where it fails to continue.
DAVE SMART: You see where it fails,
so then when you're running through the live test again,
you can see exactly the point that something's not happened.
And you can ask them, well, how does--
you call in your main content from API/cats.
MARTIN SPLITT: Whatever, from something.
DAVE SMART: And it's not coming through.
You say, well, how exactly does this happen?
What function does that happen in?
And tracing it through that way, you
can find the point it falls over.
And then, hopefully, the devs will have a good, clear path
of where to go and move there.
MARTIN SPLITT: Fair enough.
[HARP PLAYING]
DAVE SMART: So in the first case,
it was quite simply a case of moving all the calls
to the APIs and everything out of that whole registering
of the service worker logic.
MARTIN SPLITT: That's pretty cool.
I like that you're basically approaching it
from a perspective of, ooh, a murder has happened.
Let's solve this together.
And then you team up with the developers
because I've heard other people having a different strategy
than yours, where they're basically trying to figure out
what the problem is and then going to their developers.
DAVE SMART: It can work.
The problem is, with a lot of modern sites
these days, they're built, and they
go through something like webpack, the JavaScript
Obfuscator.
MARTIN SPLITT: It's a whole pipeline.
DAVE SMART: So you can't just see, oh, it's this function.
So at that point, you probably don't
have the tools and the information
to be able to say exactly where it's failing, just the outcome.
[HARP PLAYING]
MARTIN SPLITT: And looking at a site that
was partially indexed, I guess that must
have been even harder to debug.
DAVE SMART: It was because a good deal of the stuff was--
MARTIN SPLITT: Was there.
DAVE SMART: And pages were in indexing.
Unfortunately, it was a little block of information.
But it was quite an important block of information.
One of the things, one of the superpowers, service workers
give you is to work offline.
And this was information that was relevant for people
going out and doing jobs.
The devs thought, what a great way--
MARTIN SPLITT: To make that available offline.
DAVE SMART: Which it does.
MARTIN SPLITT: Good thinking.
DAVE SMART: And what they also thought is,
we want to get loads of it.
So they overtook the normal process of fetching
and did it purely within the service worker.
MARTIN SPLITT: I did that in the past.
Yeah, not going to lie.
I feel for the devs here.
DAVE SMART: Yeah, it was rational, and it made sense.
And it worked for the end user.
MARTIN SPLITT: Yeah, for the end user, just unfortunately
not for search.
DAVE SMART: Yeah, exactly, unfortunately not for search.
[HARP PLAYING]
MARTIN SPLITT: And how did that manifest in--
so you said you used Search Console and specifically
the Coverage report?
DAVE SMART: Yeah, you could see, hey, these guys
are getting covered.
They're getting indexed, but they're just not
surfacing for things they really should surface for.
Someone went in and did a search for it-- a bit of this content.
MARTIN SPLITT: And it just didn't turn up.
DAVE SMART: It was quite unique.
Just didn't have visibility.
MARTIN SPLITT: And then from that point where you see,
so it is crawled--
because if it's not crawled, it can't be indexed--
it is indexed as well.
It's just not serving in search results.
How did you then figure out where to go next?
Did you then head over to the devs,
or was there a step in between?
DAVE SMART: The first step before you
head over to the devs is, you just go,
are you just not ranking very well for this?
Is it ranking badly?
Have you got a lot of competition?
Because not everything's technical in that respect
of ranking.
MARTIN SPLITT: Yeah, true.
It might be a content issue.
DAVE SMART: Yeah, exactly.
So we had a look at the rendered HTML.
If you look in Search Console, you see--
MARTIN SPLITT: I was about to say, where's the rendered HTML?
DAVE SMART: And use the URLinspector tool.
Don't do the live test, just the first step of it.
And this nice little panel-- no pretty pictures,
but that's not really what matters.
But you can have a look.
Is the HTML there?
And sure enough, no sign of that.
So that then gives you something to go to the devs with.
MARTIN SPLITT: I think you can even
search in the rendered HTML, and then you
type in the same unique content.
And you're like, wait a minute.
This can't show up because it's not here.
DAVE SMART: Yeah, it's not there.
So then that gives you a clear path to go back to the devs.
And basically, you repeat that process.
How does that get from there?
And it's just breaking down steps.
MARTIN SPLITT: That's nice.
That's interesting.
[HARP PLAYING]
There's one thing, one opportunity,
that I think we haven't talked about,
and that is actually the developer tools in your browser
because you could go into the network inspector
and try to find out, where does this piece of content
come from?
And I think there it says who or what initiated it,
which then you need to go to the devs with because then it's
going to be like, app.min.js, line 1, which is
a huge line of spaghetti code.
And you're like, OK, great.
DAVE SMART: Plus, you can get a little bit muddy
with service workers too because your report, but a lot of stuff
is coming from the service worker.
MARTIN SPLITT: That is true.
And then you're like, hmm.
DAVE SMART: It gives you a good pass.
MARTIN SPLITT: You could use the developer tools in the browser,
as well, to disable service workers
if you have a suspicion that it's the service worker.
It's a tricky one, probably.
DAVE SMART: Yeah, you just need to go in DevTools.
If you go to the Application panel--
MARTIN SPLITT: It is hidden, yeah.
DAVE SMART: You might need to search to find
that in the first place.
And then there is bypass as well if you find a service worker.
It's a little tick box, Bypass.
Actually, it's Bypass on this Network.
MARTIN SPLITT: So it always goes on network.
DAVE SMART: Yeah, so it's just the network
that you're on at that time.
It'll then start bypassing the service worker.
But you get a great idea of what the site would
look like when the service worker is not servicing.
MARTIN SPLITT: When the service worker doesn't register
or it doesn't work.
Aha.
Interesting.
[HARP PLAYING]
To recap, you had a client coming in to you-- actually,
two clients coming in with a site that
wasn't indexed at all, and then one was indexed
but was missing content.
And both of that, you could trace through Google Search
Console and the URL Inspection tools
to be an issue that the devs needed to look at.
And then you teamed up with the devs
to actually work through that.
How did the dev part of the process work?
So you went to the developers and were
like, where is this coming from because it's missing?
DAVE SMART: Exactly.
So you just got to explain, step by step, which way--
well, our function calls that, and then
what calls that function?
And you trace it back to the origin
point of someone coming in.
MARTIN SPLITT: And then that's where they
can work from and fix that.
I guess that should be not too hard to fix, in some cases.
Well, in some cases, some case.
DAVE SMART: And certainly, in the first case,
it was a very easy fix.
It was literally just-- didn't function.
MARTIN SPLITT: Does not work for the-- yeah.
DAVE SMART: The second case, there
was a little bit more finagling that I needed to do.
MARTIN SPLITT: I cannot imagine.
I've done weird things as well with service workers.
Service workers are great but also
a dangerous tool if you're holding it wrong,
as Gary would say.
Am I holding it wrong?
Awesome.
Dave, this was a wild ride through the history
of tracing a bug in the service worker--
or with service workers in Googlebot.
Thanks so, so much for being here with me today
and telling the story.
And for all of you out there, do let
me know in the comments, what was your weirdest experience,
and how did the developers respond
to it when you were asking them about what's going on there?
And leave us a like, and subscribe to the channel
to see upcoming episodes and other content that we put out.
Stay safe, take care, and bye-bye.
Not creepy.
How dare you?
DAVE SMART: Fairies are real.
MARTIN SPLITT: Fairies are real!
[MUSIC PLAYING]