Hello in this short video we're going to show you how to add a robots.txt file to your site
This is an important step in the publication of any website
as this file tells the search engines what to index on your site and what not to index
which pages to access in which to ignore
this includes private content or
unnecessary elements that would otherwise suck up bandwidth and increase
load time on your server for no goodreason
Not having a robots.txt file can
also generate errors in your Apache log when search engine spiders request a
robot's file and don't find one
after we run our website on SEO site checkup
We can see that we are missing our robots.txt file
so when a search engine like Google sends a spider to index the contents of our site
it will try to index everything
now you might say what's wrong with that isn't it better to have everything indexed
well no actually
it's much much better to
optimize what the search engines are indexing to have them only include high
value content in relation to your target audience
It's also important for better
security to leave some parts of your site unindexed
and there might be cases
where you don't want visitors finding certain pages unless they've performed a
certain action on another page on your site first
so let's click how to fix
which will show us how to format a robots.txt file
we'll start up a new text file in a text editor
and let's go ahead and copy this
and paste it
the formatting is as simple as this
If we like we can specify which
search engine specifically should adhere to our following no index directives
so we could have different rules for different search engines if we want to
or with the asterisk we're telling all search engines to follow this set of rules
anything we don't want indexed we just include after a disallow colon
this can include certain pages such as our thank you for
your message page that visitors are redirected to after they send us a
message via our contact form there would be no value in having search engines
index this in fact since that page doesn't contain any keywords or content
related to our website's general subject this would be diluting our SEO
also we only want visitors coming to this page after they've written a message
so by disallowing the search engines to index it visitors won't come here by mistake
or perhaps there's a certain page and
certain information that you want to give to visitors only after they've
submitted their email address or signed up for your newsletter
Then you'd want to have these pages in your robots.txt file so that users can find that page
only after a certain action
We can also tell the search engines not to index
entire folders such as our images folder
but if we do that we can also allow
certain files or folders within that images folder to be indexed by simply
adding an allow command
So while search engines in this case
will not index the images folder at large within that image folder they will
index my race car image
or my portfolio images folder within my main images folder
you get the idea
let's also disallow indexing of folders that
contain our CSS our JavaScript and so on
This will not only help boost security
but again it will also optimize the content that the search engines are
indexing thereby boosting the value of what's being indexed
Then once we're done
we'll simply save the file as robots.txt all in lowercase
and then we'll upload that file into the root directory of our website
and now we're set
Just be sure to remember to update this file if you make changes to your
website and have new content that you want indexed or ignored by search engine BOTS
and lastly it's also a good idea to test your file for any syntax errors
with the google webmaster robots.txt tester tool
You can find it easily by searching for google robots.txt tester
and clicking on this support article
click open robots.txt tester
and click test
if you get the grayed out allowed status that means you had no errors in your file
and if you do return errors go ahead and fix them
feel free to check out our other videos
so you can keep fine-tuning and getting the most out of your SEO and click
subscribe to stay updated with new tips and tricks to get you to the top of the
search engine results pages