hey everybody hope you're
good today I'm going to show you a quick trick to get search engines
to not index a specific page
or pages on your website
now at first glance this concept might seem a bit bizarre
I get it most of us think that strong SEO means
indexing and getting as much as possible out there which in turn
boosts your search engine Authority
so it's kind of counterintuitive to think that there's any reason not to index
content but sometimes it's definitely a good choice to make
for example let's say that you're targeting a certain keyword or keywords
and most of the pages on your site do an excellent job of employing that keyword now obviously I'm talking
about in an organic non stuffing kind of way but maybe you have a page that's not
about that keyword set like a thank you page after your customer buys something
or after they order a subscription or signs up on your site something like
that and maybe it's a great Thank You page heartfelt and detailed maybe it
inspires the user to take further action that helps you out oh that's awesome but
maybe that page isn't about targeting your keyword sets so if the search
engines are crawling it even with a robots.txt file that tells them not to
have it pop up randomly in search results they'll still index it and
include that content in their algorithms and maybe that can actually set you back
and dilute your SEO work that's one example or maybe you've got a
members-only page that gives the user some extra info that they didn't have
before after they've submitted their contact details then obviously you don't
want that page showing up in search engines so that people can get that
extra info without submitting their contact details situations like that or
maybe you've got some terrific content on your site and you're so web savvy and
done so well with building your site that you've also included a
printer-friendly version of that same content great for the user and you've
got a slick site but the search engines can see that and go hey wait a second
this website has all this duplicate content
so let's penalize them for that or in plenty of cases let's say you have some
images on your site then those images can be assigned unique URLs by the
search engines and then pop-up and image search results searches that don't have
anything to do with you or your site and now you've got your images popping up in
results maybe that's cool anyway we can fix all that easily
now bear with me for a second this is going to look freakishly complicated but
we'll get through it first right this line of code in the head section of the
page you'd like to not have indexed then that's it
super simple actually that's really all there is to it just slap that line of
code into any page and you're telling the search engines to crawl the content
to go ahead and assign page rank value but not index it and have it show up in
search results just bear in mind that like with a lot of these changes it's
going to take some time to propagate the search engines won't know what to index
and what not to index until they've crawled your site again and see this no
index tag now while we're on the subject you might have heard about two similar
concepts the nofollow tag and the disallow directive these also have their
uses and you may find that they are exactly what you're looking for in some
cases instead of no index now we have two more videos that go into depth on
each of those so feel free to check those out but for now these concepts are
similar but quite a bit different so let me just quickly break down those
differences just like no index the disallow directive within your
robots.txt file it tells search engines not to crawl certain pages or
directories of your site now what's particularly convenient about this
method is that you can apply it to entire folders in your website or to all
files of a certain type or with URLs containing specific keywords but even
this isn't foolproof like let's say that I disallow the indexing of my page
member benefits I don't want this indexed and showing up in search results
well if I have an internal link to that URL from another page on my site or if
an external website out there links to that member benefits page then the
search engines can find it that way and still might end up indexing that URL
also with the disallowed directive search engines like Google might be
likely to lower or drop that pages ranking so to play it safe and guarantee
that a certain page or pages are not indexed and will not show up in search
results one must not have the disallow directive in robots.txt so that search
engines are allowed to crawl the page then by adding a no index meta tag with
the HTML of your page or your pages you will tell the search engines sure go
ahead crawl the content to the page to see what's there give me credit for all
the positive PageRank links and content I have there but please don't index it
to make it appear in search results and again in order to use this no index meta
tag the page you're using it on can't be blocked in your robots.txt file because
search engines will see that first then ignore the no index tag on that page
because they won't see it in the first place now also one can apply the no
index directive to a page pages or even entire folders of your site within the
robots.txt file but apparently this isn't a very reliable way of handling
the issue when done on its own according to Google's own John Mueller
without him actually going into very much detail about it recently warned
against it so it's more highly recommended to apply the no index tag
within the HTML of a page or pages so generally speaking the no index tag is a
more reliable choice now that said if you have a new website that hasn't yet
been crawled and indexed by search engines and if you have pages or folders
that you'd like to remain out of the index then using the disallowed
directive can be much faster to implement since you can apply it to many
files or folders at once so if you're fairly confident that there's no good
reason that other websites will be linking to that unindexed content then
you should have no issues but if your site's been up for a while and the pages
you want out of the index have already been indexed then use the no index tag
and then you'll have to wait until those pages are crawled again and it's only at
that point that search engines will know not to index them anymore then lastly
let's look at one more scenario so let's say you have a page with some great
outbound links to high authority sites so you've got some good PageRank signal
flow there and you want the search engines to crawl it and reward you for
it but you don't want that page showing up in search results and unfortunately
you also have some other links on that page that might be damaging to your page
rank like you've got a comment section or a section for paid content and you
can't verify that the content that shows up there will be good maybe spam bots
hit your comment section from time to and you don't wanna be penalized for
those links or maybe you're doing a link exchange and having websites put your
link up in exchange for you putting theirs up they've got to be careful with
this because search engines are always on the lookout for black hat techniques
and shoddy link schemes but let's say that you're doing this and yet you don't
fully trust the links that you're putting up or you're afraid that they're
low authority and might do damage to your PageRank not a problem in that case
just use the no index tag in the head section so that the search engines see
it and say ok we'll crawl this content and look at the links you've got there
but then apply the nofollow tag specifically to any links you don't want
to vouch for then the search engines will follow the good ones and not follow
the bad ones so to sum it all up here if you've built a brand new website that
hasn't been indexed yet and you want certain pages shut off from the search
engines entirely you don't want them indexed to show up in results you don't
want any positive or negative signal flow and you're pretty confident but
there's no reason that other websites will be linking directly to these pages
then the disallowed directive within your robots.txt file is a good way to go
for pages that you do want search engines to crawl and award positive
PageRank signal flow to but you don't want those pages to show up in search
results then use the no index tag in your head section of the HTML for each
of those pages and if you have certain links within a page that you don't want
to be penalized for like links to user-generated content that may be
spammy but you do want other content on that page to be indexed and to retain
positive signal flow then use the nofollow meta tag on those specific
links that you want the search engines to ignore and if you don't want that
entire page showing up in search results again just apply the no index tag in the
head section great hopefully you're clear on the similarities and
differences of all three and know which one to use to suit your site specific
needs feel free to check out our other videos so you can keep fine-tuning and
getting the most out of your SEO and click subscribe to stay updated with new
tips and tricks to get you to the top of the search engine results pages