(upbeat music)
- [Dana] Robots.txt files are simple text files
that live in what's called the root of your site.
You can check if your website has one,
by going to your domain.com/robots.txt.
Many CMSs, such as WordPress will automatically create one,
but that doesn't mean it's set up
quite the way that it should be.
Robots.txt files, tell search engine crawlers,
not just Google, but all search engine crawlers,
what pages or files they can or cannot access
on your website.
These are useful to keep robots out of areas
where they wouldn't find anything useful,
such as the WP admin directory and WordPress.
Your robots.txt file contains disallow, allow
or both types of instructions.
The basic format of a robots.txt file is like this,
user agent, name of the user agent,
and then disallow the thing you don't want crawled,
you only really need these two lines to make up the file.
It just needs to include what search engine crawlers
you want to listen to your file
and then what you want them to do or not do.
You can also add in multiple sets of user agents.
Let's say in this example,
you want Google to crawl one thing,
but you don't want bing to look at it,
that's what this would look like.
The first instruction we're telling Googlebot,
look at whatever you want.
And the second instruction we're saying
bing, don't take a look at this subdirectory.
Also you'll notice that I used a relative folder
for the disallow instead of the full URL.
You want to make sure not to use full URLs
for these types of commands.
You might also see robots.txt files that look like this,
where you're saying user agent star, disallow everything.
User agent Googlebot allow everything.
What this file is saying is that only Google
is allowed to crawl the site
and none of the other crawlers are allowed to do so.
Again, some crawlers might just go ahead and ignore this
and crawl the site anyway.
The allow command that I used in this example
specific to bingbot and Googlebot,
other robots don't pay attention to that command,
they only acknowledge disallow commands.
Almost every site has a robots.txt file.
You don't absolutely need one, but it can be useful.
I recommend that every site should have one.
Since most sites have one,
and it's a file that needs to be accessible
to the search engines,
that means that anyone can see
anyone else who's robots.txt file.
Try looking at other websites robots.txt files,
to see what they've done.
When you're building your file, you also want to make sure
you don't have the conflicting rules.
This can get tricky if you have a really long file
with lots of directives.
For examples of a really long robots.txt file,
take a look at a site like Buzzfeed,
where they have lots and lots of rules going on.
You'll see up here that they have this crawl delay directive
and I highlight it for you.
It's right after user agent msnbot,
which is now called bingbot.
This is used to tell the crawler
to wait a certain number of seconds
before crawling the next page.
Google ignores this because they want
you to set the crawl rate in Google search console,
but being in Yandex will respect this directive.
You'll also see in this file that they use a lot of stars
and dollar signs for some of their directives.
A star is a wild card,
and that can be used to match multiple URLs at once.
So for example, if we look at this file,
you can see here that they have this disallow command.
Star and then question mark, S equals mobile.
What they're saying here is they don't want the crawler
to look at anything that ends with
question mark S equals mobile.
That dollar sign command indicates the end of the URL.
Let's say, for example,
you didn't want any PDFs on your website to be indexed.
This is what that would look like.
You're saying any user agent
disallow any file name at all if it ends with a PDF.
That dollar sign indicates the file name ends after PDF,
so if a file name is called PDF X,
then it might still get indexed.
You can also include comments in your robots.txt file.
Just start the line with a pound symbol,
but make sure not to continue on to a second line.
Every line that you want to have commented out,
shouldn't have a pound symbol in front of it.
One last thing you'll want to do
is linked to your sitemap file
at the beginning or the end of your robots.txt file,
like we've done here on the Kick Point website.
I typically do it at the end,
but some people prefer the beginning.
It's really entirely up to you.
Google will probably already have your site map
since you searched submitted in Google search console
and being in their webmaster tools,
but other crawlers will also find this information useful.
A couple of last notes on formatting your files,
make sure that the file name is all lowercase.
You can't have robots.txt with a capital R,
all caps, robots.txt, or the file won't be read.
You want to make sure not to name a file
anything other than robots.txt,
this isn't a time to get fancy.
And keep in mind that some crawlers
will totally ignore your directions.
And unfortunately, that's pretty common
for tools that scrape pages for content for example.
I always say that robots.txt is the suggestion,
it isn't a rule.
Once your robots.txt files created,
make sure to test it using Google's robots.txt tester,
and then check for errors.
I've seen examples where an entire site
was negatively impacted by a poorly written,
an untested robots.txt file.
These tiny little files are enormously powerful,
make sure you spend the time to get it right.
(upbeat music)