CRYSTAL CARTER: OK, ready?
MARTIN SPLITT: You can.
CRYSTAL CARTER: And there and there.
MARTIN SPLITT: Ah!
[LAUGHTER]
[MUSIC PLAYING]
Hello, and welcome to another episode of "SEO Fairy Tales."
With me today is Crystal Carter, head
of SEO communications at Wix.
And when you're not working, you are "kilting--"
CRYSTAL CARTER: Quilting.
MARTIN SPLITT: Quilting?
CRYSTAL CARTER: "Kilting?"
MARTIN SPLITT: I'm not an English native.
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: I don't know how to pronounce that properly.
I only read the word.
CRYSTAL CARTER: I could make it--
I could make a quilted kilt. That's a thing I could do.
MARTIN SPLITT: So you're quilting a kilt.
CRYSTAL CARTER: No, I haven't, but I could.
I could make you one.
Would you like one?
MARTIN SPLITT: Yeah, sure.
CRYSTAL CARTER: OK.
[LAUGHS]
MARTIN SPLITT: Be aware it's on camera now, though?
So that's a promise.
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: Fair enough.
Or you're in a canoe.
CRYSTAL CARTER: Yeah, well, I am occasionally in a canoe.
There's a place near me where you can go canoeing in a canal,
and that's what we do sometimes.
MARTIN SPLITT: That sounds like kind of tongue-twister,
like canoeing in a canal.
CRYSTAL CARTER: [LAUGHS] Right?
MARTIN SPLITT: Kilting a kilt?
Quilting a kilt?
Whatever.
Let's talk about something that I can actually
pronounce properly, which is technical SEO.
[MUSIC PLAYING]
So you brought me an interesting challenge
that you experienced and wanted to talk about that.
So what did you bring for us today?
CRYSTAL CARTER: Yeah, so I wanted to talk to you
today about an indexing challenge
that I had from a client.
And they were looking at their--
we were looking at their web situation,
and we had a look at Search Console.
We could see that they had something like 80,000-ish URLs
that they had in their sitemap, but they were only indexing
about 55,000.
And it was an E-com site, so they
needed to make sure that their products were actually
available for people to see online.
[MUSIC PLAYING]
And so I started where most good SEO starts,
and I started with a crawl.
And I had it set up on a few different tools.
So I had it set up on ContentKing and on SEMRush,
and I had it set on the Ahrefs.
But what the situation was, it was
because it was such a big site that the config that I
had for those particular tools wasn't set up
for a regular crawl.
It wasn't catching the issue with the crawlability.
It said there was a crawlability issue,
but everything I diagnosed wasn't really--
MARTIN SPLITT: Was fine.
CRYSTAL CARTER: --filling the issue.
So I did what everyone does, and so I went to Screaming Frog.
And my trusty little laptop, which had never,
ever failed me in all my days, just
could not complete this crawl.
It just couldn't do it.
And I thought-- at first, I thought
it was something wrong with my laptop,
or something wrong with the crawl settings,
or something going on there.
And then I realized that what was actually going on
was that there were redirect chains.
[MUSIC PLAYING]
And a redirect chain is when a link--
is when someone clicks on a link, or someone crawls a link,
and it goes to a link, which is redirected
to another link, which is redirected
to another link, which is redirected-- and on,
and on, and on, and on.
And this is what was happening with the crawler.
And so I was looking at the-- and there's a little chart that
shows you where the 301s are.
And it just kept getting bigger, and bigger, and bigger,
and bigger, and bigger, and bigger, and bigger.
MARTIN SPLITT: [LAUGHS] And eventually, it's
just like, nah, I'm good.
CRYSTAL CARTER: Right!
I was just like, let's stop this.
What is going on here?
And so I looked into it, and I found
that there was only about five URLs
that all of these redirects were coming from.
And it was a couple of URLs which had some broken images.
And the images were redirecting to redirects,
redirects, redirects.
MARTIN SPLITT: Effectively breaking them, yeah.
CRYSTAL CARTER: Effectively breaking them.
So I couldn't complete the crawl.
And all I could think was, if I can't complete the crawl,
then Google can't complete the crawl, and nor would they--
nor would Google even--
they'd just look at this and go, this isn't working.
We'll just keep going somewhere else.
[MUSIC PLAYING]
So the actual fix for this was pretty simple.
The fix was, remove the links.
I have a saying sometimes with-- people
say, oh, how do I fix this 404?
And then sometimes--
MARTIN SPLITT: Yeah, it's just--
CRYSTAL CARTER: You can just rid of it.
MARTIN SPLITT: Yeah.
CRYSTAL CARTER: Like it ain't broke if it ain't there.
MARTIN SPLITT: Exactly.
[LAUGHTER]
CRYSTAL CARTER: So I was just like,
we need to get rid of these links.
But the actual-- in order to diagnose
this, what we had to do is we had
to crawl until we got to a stop.
And then, I'd have to exclude that URL
and then crawl it again till I got to another stop
and then exclude that URL, and just add them all to a list.
And in the end, we were able to get there.
But yeah, it was tricky.
And the client, the client didn't
believe that this was an issue, but I was like, I assure you,
it is.
And after we cleared all of these redirect chains,
we saw the indexation started to go up
because Google was actually able to finish
the crawl in the same way that I was actually
able to finish the crawl.
MARTIN SPLITT: Surprise!
Yeah.
Infinite crawl space is fun times.
[MUSIC PLAYING]
But you said you set up a crawl in different tools.
Why did you do it in different tools,
and how did you set that up?
CRYSTAL CARTER: So one of the things that's important
is all of the tools-- and tools are great.
Google Search Console is great.
And all of the tools that I listed before are fantastic.
They all have different criteria.
They all look at different things.
So it's really important, if you're
trying to diagnose an issue, to get a second opinion
and to check them across various different tools.
Generally speaking, if you're trying to verify a tool,
if you're right on all of them, then you're good.
[LAUGHTER]
But if you see one flag on one and not on another,
then you might need to get even another opinion.
You might need to get--
MARTIN SPLITT: A third opinion.
CRYSTAL CARTER: A third opinion.
So that's why I did that.
And also, sometimes, you can set your crawl--
particularly on a third-party tool,
you can set them so that they only
crawl a certain number of URLs.
So if you have a free version of a site--
of a third-party tool-- you might have a max number
of URLs, for instance.
MARTIN SPLITT: And you don't get the full picture
at the beginning, yeah.
CRYSTAL CARTER: You don't get it, you don't get it.
And so you might say, oh, I've got 100% on that audit tool.
But you've only crawled 100 URLs.
And if that's not a good representation of your URLs
overall, then that's not really going to be very helpful.
So you want to-- when you use multiple tools,
it gives you more data.
MARTIN SPLITT: That makes sense.
That makes sense.
[MUSIC PLAYING]
And how did you set up the tools so that--
I mean, did you have a hypothesis
that you started with when you saw like, hmm,
something's missing here?
CRYSTAL CARTER: Yeah, so I went through Google Search Console.
Google Search Console will tell you what is indexed
and what is discovered.
MARTIN SPLITT: Yeah.
CRYSTAL CARTER: And they'll tell it--
and you can see the difference.
So if you're seeing a big difference between what's
indexed and what's discovered, you
can have a look at those URLs and see
where some of the issues are.
For this particular client, there were some other issues
as well.
So they had some particular page types
that should have been no-indexed,
and they needed to do that.
And when we did that, that helped.
But they also had-- they also had these redirect issues
that were coming in.
MARTIN SPLITT: Yeah.
CRYSTAL CARTER: So if you're trying
to address why some of your URLs are discovered
but not in index, it's important to look at these things.
So yeah, I would say Google Search Console
is a really good first port of call
to see what URLs they are indexing
and to discover any trends.
So if you're seeing a trend where URLs in a certain folder
are not being indexed, then there's
probably something wrong with that folder.
MARTIN SPLITT: True, exactly.
Why not the rest of the site?
Why is it in this part?
There must be something-- well, makes sense.
That makes sense.
CRYSTAL CARTER: Right, exactly.
So it could be that there's something
in the robots.txt that goes to that particular folder,
or it could be that there's a bug on that [INAUDIBLE]..
MARTIN SPLITT: It's interesting that you said like, oh, yeah,
then they fixed the no-indexing.
Like they no-indexed things that shouldn't have been there,
and then it got a little better for them--
which makes sense because you free up
crawling resources for that.
But then, you still run into the redirect chain.
CRYSTAL CARTER: Right.
MARTIN SPLITT: So that makes sense.
[MUSIC PLAYING]
All right, so then you know--
OK, did you see a pattern at that point?
Or was it more like, everything is screwed?
CRYSTAL CARTER: Yeah, so the pattern
that I was seeing for that particular one
was that there were certain page types that kept showing up
as being [INAUDIBLE].
MARTIN SPLITT: Right, and so it might
be something in the template of these page types?
Oh!
CRYSTAL CARTER: Right.
Right, so there was something there.
And then, the other thing that I--
but also, the thing that was fundamentally a problem to me
was that I couldn't finish the crawl.
Like-- [LAUGHS]
MARTIN SPLITT: I mean, yeah, OK, fair enough.
CRYSTAL CARTER: And you shouldn't
be able to finish-- you should be able to finish the crawl,
even if it's a big site.
It might take you a little while longer than a smaller site,
but you should be able to finish the crawl.
And if you can't finish the crawl,
then Google can't finish the crawl.
MARTIN SPLITT: How would Google-- yeah, fair.
CRYSTAL CARTER: And nor would they want to,
because if you were asking for--
MARTIN SPLITT: And it is hard.
CRYSTAL CARTER: --anything from anyone,
and they just kept giving you the runaround-- they say,
oh, talk to this person, talk to that person,
talk to this person, talk to that person,
and then, eventually, you give up.
So it's important to make sure that you've
got-- that your crawl is accessible,
that your crawl is smooth, smooth sailing.
So there shouldn't be any redirect chains.
There shouldn't really be any internal redirects
if you can avoid them.
Like, why are you doing that to yourself?
MARTIN SPLITT: I mean, if you have like a migration that you
need to deal with, or something has been decommissioned,
and something else steps in, and you for some reason can't keep
the URL structure-- because that's what you should do--
I see why you might use that as a tool.
But having them on accident is not--
CRYSTAL CARTER: No, It's not so good.
It's not so good.
[MUSIC PLAYING]
And it can compromise you in a number of ways.
So the redirect chains and 301s, just in general,
if you have too many of them, can slow down
the page experience for users.
MARTIN SPLITT: Yeah.
CRYSTAL CARTER: So when someone clicks on that link,
they have to be redirected to another like.
And all of that time, they're waiting.
MARTIN SPLITT: And eventually, the browser gives up.
CRYSTAL CARTER: Right?
[LAUGHS] Exactly, exactly as well.
MARTIN SPLITT: [LAUGHS] The browser's like, you know what?
We tried this.
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: Right?
We really, really wanted you to have the experience,
but someone really does not want you to see anything here.
CRYSTAL CARTER: Right, that poor little dinosaur,
he did his best.
[LAUGHS]
MARTIN SPLITT: Such a good thing.
The dinosaur is such a good feature.
CRYSTAL CARTER: It's so good.
It's like, oh, the internet's off,
but at least I get to jump over the cactuses!
[LAUGHS]
MARTIN SPLITT: Exactly, exactly.
Which is very similar to the process of debugging
technical issues sometimes.
You've got to jump over these cactuses.
CRYSTAL CARTER: Right!
[LAUGHS] Exactly!
MARTIN SPLITT: Awesome.
[MUSIC PLAYING]
Well, and then you figured out, OK, so it's this page type.
And then, how did you figure out that the redirect
chain was happening?
CRYSTAL CARTER: Right, so what will happen,
and what's actually happening-- and I was looking at this.
So Screaming Frog is very much like a spreadsheet.
MARTIN SPLITT: Mm-hmm.
CRYSTAL CARTER: The user interface
is very much like a spreadsheet.
And what happened was I was looking at this,
and I was like, OK, let's have a look at these redirects.
So you go into the tab that says Redirects.
And then I could see that it went 1, 1, and then 2, and then
2, and then 3, and it was just filling up.
It was like, redirect, redirect, redirect, redirect.
MARTIN SPLITT: Right, that waterfall
there should not exist.
CRYSTAL CARTER: Exactly!
MARTIN SPLITT: Right.
CRYSTAL CARTER: So I think as you go through that,
you need to--
if you see that kind of thing, that's a big, big red flag.
MARTIN SPLITT: Yeah.
CRYSTAL CARTER: And then, I had to check
that, no, one of-- the client was like, how do I
find these in the future?
I'm like, well, we can find them in the future.
So some of the tools that I set up,
they'll alert you if you have a new 301.
But we had-- but these were historical ones.
So we had to just go through them
individually and find them.
But yeah, it's-- and I've seen it before where sometimes,
these can cause server errors as well, which has another--
that's another knock-on effect.
Google is seeing-- within Search Console,
there's a report that will tell you if your server is dicey.
[MUSIC PLAYING]
[LAUGHS]
MARTIN SPLITT: It's going to be like, we're not
going to topple your server.
CRYSTAL CARTER: Right!
MARTIN SPLITT: We're just going to be touching it carefully.
CRYSTAL CARTER: [LAUGHS] Exactly, or just--
MARTIN SPLITT: Light.
CRYSTAL CARTER: Yeah, so if they're
saying, if Google's saying that, then it's
like you're not as reliable a source as you could be,
and there might be other sources that are more reliable.
And so you just kind of need to get your act together.
So it's really all it comes down to.
MARTIN SPLITT: It's not even about that.
It's more like, so we're in this weird dilemma that
is like, on one hand, we want to crawl as much as we can
as quickly as we can.
On the other hand, we don't want to eat up all your resources
so you can't actually serve your actual customers because that
would be kind of not nice.
CRYSTAL CARTER: Right.
MARTIN SPLITT: And whenever we are
seeing like your server goes like, whoa, whoa, whoa, whoa,
whoa, whoa, whoa, whoa, I need a minute here,
then we're like, all right.
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: We'll slow down a bit, no worries.
Which is probably not what you want to happen.
CRYSTAL CARTER: No.
MARTIN SPLITT: Because especially
if you have millions of products, you're like,
no, no, no, no!
Don't!
Come back, come back, come back!
CRYSTAL CARTER: This is a flash sale!
We need this-- yeah.
MARTIN SPLITT: Let's do this, quick, quick, quick.
And we're like, yeah, but when we do that, then--
eh.
So yeah, it's in your own interest to fix that.
That's true.
CRYSTAL CARTER: Absolutely, and I've
seen that happen on the other end--
not to do with redirect chains, but I've
seen it where someone had--
they were like, oh, we want to do a big Facebook campaign
thing.
And so they did this big, big Facebook campaign, sent loads
of traffic, and as soon as they sent it to them,
the site crashed.
MARTIN SPLITT: The site crashed, right.
CRYSTAL CARTER: And they were like-- they're like, oh,
what can we do to optimize?
I'm like, sort out your server.
And they were like, oh, well, but we want to do this.
And we want to have this picture on the ad.
And I'm like, you have to sort out your server.
There's literally-- you're just throwing money
at nothing at the moment.
MARTIN SPLITT: Yeah.
CRYSTAL CARTER: And in fact--
MARTIN SPLITT: People click and go to nowhere.
CRYSTAL CARTER: Exactly, and that actually does worse--
more damage for you than just not having the sale
in the first place.
[LAUGHS]
MARTIN SPLITT: [INAUDIBLE].
[LAUGHTER]
Yeah, oh my god, OK.
[MUSIC PLAYING]
And you saw all these redirects.
And how did you identify where they were coming from?
Because you said it was very few URLs that caused this.
CRYSTAL CARTER: Right, so when you're
in it, when you look at the crawler, you'll normally see--
whether you're using a third-party tool,
or whether you're using other things,
they'll tell you that there is a redirect on that page.
And then, you have to go to that page and find--
MARTIN SPLITT: --whatever redirect it is.
How do you find that, then?
CRYSTAL CARTER: Yeah.
So there's a few different tools that you can use.
On this particular one, it was an image.
It was an image that had that little broken image icon on it.
MARTIN SPLITT: And you're like, hmm, I wonder!
CRYSTAL CARTER: Yeah, exactly.
And I think that the thing that was also funny about it
was that these were not high-value pages.
They were just-- these were older products and--
MARTIN SPLITT: Oops, yeah.
CRYSTAL CARTER: So exactly, so they were just
messing up the whole crawl just by existing.
So we were able to remove the--
so there, we were able to see the images.
But there's other tools that you can
get that will tell you if you have lots of 301s,
or lots of 404s, or lots of other crawl error
issues on your pages.
So there are a few Chrome extensions
that you can get that you can use to address pages.
So there's a Chrome extension that I
use that, when you click it, and it will tell you
all of your 301s, and all of your 200s,
and all of the everything.
And you can see it right there.
And that's a good one if you're not like a technical SEO who
has all of these licenses, or if you're not generally
looking at this [INAUDIBLE].
MARTIN SPLITT: The browser has it built in.
You can also go into the Developer Tools,
open the Network tab, and then you see, ooh, interesting.
CRYSTAL CARTER: Exactly.
[INAUDIBLE]
MARTIN SPLITT: That's not where should come from, yeah.
Mm-hmm.
CRYSTAL CARTER: Exactly.
MARTIN SPLITT: Ah, That's nice.
All right.
[MUSIC PLAYING]
And then, you've removed the images,
and the problem went away?
CRYSTAL CARTER: Yeah, so we were able to remove--
so we removed the images.
We were able to fix the links.
And we were able to figure out-- we
were able to set up a framework where we could identify when
we were creating new 301s or creating new redirect
chains for the future.
And--
MARTIN SPLITT: How did that work?
CRYSTAL CARTER: So that's with certain tools you can get.
You can get notifications when you have issues.
MARTIN SPLITT: Ah, so you are basically crawling the site,
and then, whenever you catch one that wasn't there
before, you're like, hold on!
CRYSTAL CARTER: Yeah, so there's a couple of tools.
So Little Warden is one.
Content King is another one, where they will notify you
if things go wrong.
MARTIN SPLITT: That's really nice, yeah.
CRYSTAL CARTER: So I've had it before where I had a client,
and we had some new devs, and they were doing some stuff.
[LAUGHS]
And I literally-- I was on the school run, right?
So I'm on the school run, walking to school,
and I get the notification that it's like 500 errors,
like 7,500 errors.
And I was like, what?
And I said, devs, you broke the blog.
And they were like, no no, it's fine.
Then I was like, It's not.
MARTIN SPLITT: [LAUGHS]
CRYSTAL CARTER: It's absolutely not.
Like, I've got this thing.
MARTIN SPLITT: No, no, no, you're holding it wrong!
Like, I'm not.
CRYSTAL CARTER: No, I'm not.
I'm not.
And it was really, really stressful.
But we were able to fly it straight away.
I also had another time where a dev-- bless his heart--
I said, oh, I want this.
And I said I wanted like a new feed set up.
And he deleted my sitemap.
And I was like, what are you doing?
And he's like, oh, you didn't have one?
I was like, yes, I did!
Put it back!
What have you done?
[LAUGHS]
MARTIN SPLITT: I didn't change anything!
CRYSTAL CARTER: Right?
No, you definitely, definitely did.
Like, roll that back.
And so yeah, he rolled it back, thank goodness,
before Google caught it.
I was like, don't look!
Please don't look!
So those tools--
MARTIN SPLITT: Hear me out, there will be a site map.
I know not right now--
CRYSTAL CARTER: [LAUGHS] We're just sorting it out!
So yeah, so we were able to flag that issue.
But yeah, those-- if you're not set up on one of those tools,
a lot of them will offer you with freemium versions
and things.
And Google Search Console also tells you when they find them.
But ideally, you don't want them to find them first.
[LAUGHS] Because--
MARTIN SPLITT: Fair, yeah.
CRYSTAL CARTER: So you want to make sure that you--
so you can have tools--
MARTIN SPLITT: You can proactively monitor.
CRYSTAL CARTER: Yeah, exactly.
MARTIN SPLITT: Awesome.
CRYSTAL CARTER: And they can save you a lot of trouble
further on.
Because again, if you have a dev who's changed something,
then you can say, change it back, now,
before there have been 17 other changes that
have happened across the site.
MARTIN SPLITT: And then it might depend on one that's-- oh,
yeah, yeah.
CRYSTAL CARTER: Right, and then you
have to unpick the whole thing.
So you know, a stitch in time saves nine, all of that.
[MUSIC PLAYING]
MARTIN SPLITT: All right, yeah.
Then, we are back to quilting.
CRYSTAL CARTER: Quilting!
[LAUGHTER]
MARTIN SPLITT: I got to learn something today.
CRYSTAL CARTER: You know what?
I have a LinkedIn post that's brewing about how quilting
is very similar to SEO.
[LAUGHS]
MARTIN SPLITT: Oh, it's one of those that is like,
do you know what?
XYZ taught me about SEO.
It's fantastic.
CRYSTAL CARTER: Oh gosh.
Yeah, yeah.
MARTIN SPLITT: Anyway, before we get into quilting and canoeing
in a canal, just a recap.
So you had an issue that was brought up by a client,
and then you used crawlers to figure out what was going on.
The crawlers were experiencing problems.
Search Console gave you a good hint.
And then, you identified the cause
to be a redirect chain, which is always fun and games.
In your case, conveniently, you can just remove the image
and be fine with it.
But you also made sure that that doesn't
reoccur by setting up monitoring with a bunch
of different tools.
That's really, really cool.
That's a solid plan.
So not only did you identify the problem,
but you also work with a bunch of tools
to quickly resolve it and prevent it in the future
by using monitoring.
I think not enough people are using monitoring [INAUDIBLE]..
CRYSTAL CARTER: Yes.
MARTIN SPLITT: That's a big one.
All right.
What a wild ride that has been!
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: A whitewater ride in a canoe, I guess.
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: I guess whitewater you'd probably
do in a kayak, but whatever.
CRYSTAL CARTER: It's OK.
MARTIN SPLITT: It's a boat.
CRYSTAL CARTER: [LAUGHS]
MARTIN SPLITT: As far as I'm concerned, it's a boat.
The only thing left for me to say
is, thank you so much for telling me that story.
Thanks a lot for being here today with me.
And it's great to have you on the show.
CRYSTAL CARTER: Thank you so much, Martin.
MARTIN SPLITT: And for all of you out there, thanks a lot
for watching, again.
Do leave us some comments on weird issues that you found
and which tools were not helping or helping you to resolve them.
Also, please like and subscribe if you want to see more of us,
and stay safe, take care, and have a great time.
Bye-bye!
CRYSTAL CARTER: Boosh!
No?
MARTIN SPLITT: I don't know.
I was not prepared.
But I could do whatever.
Like, what happened?
CRYSTAL CARTER: [LAUGHS]
[MUSIC PLAYING]