Hello, in this short video
we’re going to show you how to add a robot.txt file to your site.
This is an important
step in the publication of any website,
as this file tells the search engines what to index
on your site, and what not to index,
which pages to access, and which to ignore.
This includes private content, or unnecessary elements
that would otherwise suck up bandwidth and increase load time on your server
for no good reason.
Not having a robots.txt file can also generate errors
in your apache log
when search engine spiders request a robots file, and don’t find
one.
After we run our website on seositecheckup, we can
see that we are missing our robots.txt file. So when
a search engine like Google sends a spider to
index the contents of our site, it will try to index
everything. Now you
might say, what’s wrong with that, isn’t it better to have everything indexed?
Well no, actually. It’s much, much better
to optimize what the search engines are indexing,
to have them only include
high value content in relation to your target audience.
It’s also important for better security to leave some parts of
your site unindexed, and there might be cases where you don’t want
visitors finding certain pages unless
they’ve performed a certain action on another page on your site first
Hear more by watching the full video!
So let’s click how to fix, which will show us how to
format a robots.txt file.
We’ll start up a new text file in a text editor.
And let’s go ahead and copy this,
and paste it.
The formatting is as simple as this. If we like
we like, we can specify which search engines specifically should adhere
to our following no-index directives.
So we could have different rules for different
search engines if we want to. Or with the ,
asterisk we’re telling all search engines to
follow this set of rules.
Anything we don’t want indexed, we just include after a
disallow colon.
This can include certain pages, such as our
thank you for your message page that visitors are redirected
to after they send us a message via our contact form.
There would be no value in having search engines index this
in fact, since that page doesn’t
contain any keywords or content related to our website’s
general subject, this would be diluting our SEO.
Also, we only want visitors coming to this page
after they’ve written a message.
So by disallowing the search engines to index it,
visitors won’t come here by mistake.
Or perhaps there’s a certain page, and certain information
that you want to give to visitors only after they’ve
submitted their email address, or signed up for your newsletter.
Then you’d want to have this in your robots.txt file
so that users can find that page only after a certain action.
We can tell the search engines
not to index entire folders, such as our images folder.
but if we do that, we can also allow certain
files or folders
within that images folder to be indexed by simply adding an
allow command. So while the search engines in this
case will not index the images folder at large, within that
image folder they will index my racecar image.
Or my portfolio images folder within my main images
folder. You get the idea.
Let’s also disallow indexing of folders that contain our
CSS, javascript, etc. and so on
This will not only help boost security, but again, will also
optimize the content that the search engines are indexing,
thereby boosting the value of what’s being indexed.
Then, once we’re done, we’ll simply save the file
as robots.txt, all in lowercase.
And we’ll upload that file into the root directory of our website.
And now we’re set.
Just be sure to remember to update this file
if you make changes to your website, and have new
content that you want indexed or ignored by search engine bots.
And lastly, it’s also
a good idea to test your file for any syntax errors
with the Google Webmaster Robots.txt tester tool.
You can find it easily by searching for “google robots
txt tester”,
and clicking on this support article.
Click “open Robots.txt tester”,
and click test
if you get the great out allowed status that means you have no
errors in your file and if you do return error
go ahead and fix them