Hey folks today I'm gonna take you through what the
Disallowed Directive in your robots txt
file is for
and how to go about formatting and implementing it
Now else you cover different uses
for DISALLOW NOFOLLOW and
NOINDEX in another video
so feel free to check those out for more through
explanation of the differences between all three
but briefly here is the comparison in a nut shell
With the no follow meta tag applied to
specific links in a pages html code
you tell in the search engines not to
follow this link and therefore
not to assigned any page rank to them
resumably because they might reflect on your page negatively
this can applied to comments section
or paid content that might contains spamy links
or negative outbound links that you can control
with a no index meta tag
applied in the head section
of the pages HTML
here you tell in the search engine not to index
or crawl by particular page
and haven't not shown up in a search results
that are quiet a few uses for this
but often or not
this is applied to thank you page
or similar pages that are contributing positively to
overall SEO effects
over to pages you don't know the users accidentally stumbling across
when searching for something
I'll come back to this is bit more because in some cases
you want to use this method instead of Disallow Directive
like
generally speaking you can think of kind of like this
the Disallow Directive is a great method to improve SEO.
and make sure search engines are spending
their precious time indexing pages that are positive
in terms of page rank and content
and to ignore pages that aren't.
and this can be done page by page where at the folder level
or by keywords in file types
but for sensitive information
or for the pages you really really don't want to show up an search result by accident
than a playing no index meta tag page by page is your solution
just note there with both of these tags no index
and no follow
neither will apply if we use this disllow directive in robots txt file
That's all we''ll cover here today
the robot txt files lives in the root directory of your site
and if the first thing search engine spiders will check when crawling in your site
whatever you tell them they're allow to do or not allow to do when it comes to crawling your website
they'll always respond to this command first
they're typically a robot txt file starts with the line user-agent:* wild card
this is general command telling all search engines to listen up and take note of whatever
directives follow
within that robots file
but if you want to you can also get very specific here
and you can tailor all your directives or specific web crawlers
for instance i can give google bar very specific rules
i can give bing's bar very specific rules
or above
or as many as specific paths as I like
now in fiery and its a great question the people often ask
you can target certain rules for bad bots for lack of a better word
these can be malicious bots designed to crawl around
looking for security vulnerability is in your site
or just playing annoying one's that are part of some service out there in the internet waste plan
that are of no use too
And that can end up burdening your server with excessive traffic loads
poor bots running around trying to harvest email addresses
thing is though in reality these
bad bots can often the designed to flood out ignore your robots txt files anywhere
so in my opinion its not really work that it trying to rules for them
if copy is of bad bots are using multiple IP address
if also pretty hard pin them down and block there connections
but on the other hand if you find that may be there one particular bad bot thats operating from a single IP address
then you can configure your server or network firewall to block access
but for now let just set that a side and just focus on the legitimate search engine bots
and why it used Disallow Directive
with a disallow directive you're telling them not to bother crawling that page
or pages or that file or folder's of file and having them show up in search result
this is a very critical issue in SEO and it shouldn't be dismissed
you'd surprised how many times someone either makes mistake of
blocking search engines from their entire site by accident
or what's much more common
is not using this directive at all
which results in search engines wasting their precious crawl time indexing pages
that aren't contributing positively the SEO
like purely functional file
or private content
Its very important that search engine crawl your site
as he efficiently as possible
and the disallow directive in your robots txt file is your number one friend in that effort
All that said just be aware
that even with the disallow directive
it is possible in fiery that search engines can still find an index that URL
If it's linked to from other parts of your website
or from other outside sources
in essence well we robots txt file is
I don't like a stop sign coming of the high way to your house
search engines can still end up coming to your house by way of side streets
your robots file isn't intended to remove URL's from the index
It's only to prevents search engines
from crawling certain pages or folder directly
now I'll show you had to tackle that issues specifically at the end of the video
But for now lets get to it
the syntax is simple
Just disallow colon space
then the page, the file or the folder that you don't want them to look at
Now in this case with nothing after the disallow
there is no target to disallow
so the search engine will ignore this
with a forward slash
you actually preventing your entire server and all of its contents from being indexed
than that's
probably now where you're going for
If you are trying for get most out of your SEO efforts
more common uses of disallow directive are with the CSS
your Java script files
may be a private folder that you have
or backup folder
a CGI_bin that contains functionality for your website
but no SEO related content
A temp folder,
A junk folder
may be the location of your admin panel and all its related content
this are all pretty common to disallow
you can also make use of keywords
A wild card and dollar sign in your disallow parameters
these could be quiet powerful feature to leverage
but let's go were exactly how to use them
first lets start with how to use keywords
if you specify a folder name
slash
and use a keyword
lets say that we don't want indexed any URL that contains word test
then that test allow only applies to all URL that contains test
within that specific folder
if you want to applied it site wide
then you want to start with your root directory
now in this case search engines will ignore any folder name test
for that start with test
like my folder test-subject I have in my directory
It'll also ignore any HTML or PHP file named test
however so you know
unfortunately it doesn't apply to anything named test with the capital T
this is case sensitive so if we also have pages like that we don't want indexed
we'd have to make other disallow directive
and also neither of these applied to URL
that have test as a parameter
you can also disallow certain files types
For instance using a wild card in the PHP file extension
the search engine will ignore all PHP files
just note that again this is case sensitive
so do you have any upper case PHP file extensions
you'll also need to reflect that in the disallow directive
and with the dollar sign
you're telling that disallow directive to only apply to URL
that end with the PHP extension
so this would not apply to PHP file that end with a parameter
with a forward slash or php5 files for instance
and by the way it should also be noted that the allow directive
at this moment in time is only supported by Google and Bing crawlers
but you can disallow entire folder for instance
then use the allow directive to let them crawl for a specific file
or files within that folder and to ignore all the rest
If for some reason you want create this rule for all search engines out their
then the only solution is a bit on the awkward side
In that case the only thing you can really do is to have
All the content that you want disallowed
under the umbrella of one folder
then store those files you do wanna crawl in the folder one level up
Now I'm not exactly sure why someone wanna do this
It's quiet rare but
just in case that useful to you there you go
that would be the way to go about it
otherwise if the google and bing bots are good enough for you then you can go ahead and use the allow command
just be careful not to have conflicting commands
like lets say within this folder I have a sample HTML file here
If I allow this entire sample folder
But I disallow all HTML files within it
that in this case search engines won't know what to do
A rare example but something to be aware of then the last
anyway whats far more common is to disallow specific folders in their entirely
or to disallow specific pages
of course if that pages in a sub folder
than the location should reflect that
In this case buried in one of my folders
in another sub folders
are a few files i have split testing that i don't crawl since they contain duplicate content
so I'll just add those here
now quiet of few webmaster use canonical URL's or meta robot tags
to avoid been penalized for duplicate content
but that's not going to mean that search engines will still crawl back
content and allowed to show up in search results
with a disallowed directive you don't have that worry
and also on a general note the robots txt file always has to be named exactly that all lower case
and while we're at it
you'll also want to be sure to include your site map in the robot txt file
again these disallow directives are an incredibly useful tool to direct your crawl budget
to your best and brightest SEO content
to prevent search engine from indexing unhelpful content
is a great way to make sure that they are crawling indexing the right content
to sky rocket your SEO efforts
but always be careful because using error ridden or improper usage of disallow can damage your SEO
so be sure to implement it wisely
and then when all is set and done and you've created your robot txt file named it as such
and have it saved into your root directory
its definitely worth taking one last moment to test it
just google 'Google robot test'
and you'll find their testing tool designed precisely for this purpose
just open the tester
select your website
and test
and then take care of any formatting error in the event that are already
hopefully not
and finally one last word on the issue of indexing still taking place even with the disallow directive
this can happen in cases where the pages already been indexed
that there shouldn't be an issue if the website is brand new
this typically occurs in cases where the page has already been indexed
or if another website out there in the landscape links to it
or we have other links on your site that point back to that page that you don't wanna index
In those cases the search engines will hit that fist directive
lets say not to follow our contact us page
but as they're crawling our home page for instance
they become a cross a link to our contact page from there
they might very well head over and end up crawling it and indexing that URL
now i wouldn't panic about this and scrap all the disallowed directive in robots txt file just because of this
just be aware of the possibility and keep it in mind particularly
if you have a webpage with sensitive information you don't wanna indexed
or have people accidentally stumbled across in search results
In those cases you'll want to remove the disallow directive from robot txt
and add this no index meta tag to the head section of that particular page
then in search engines may come to that page through a link thinking OK this might be of interest
then see that meta tag and realized that its not to be indexed at all
and lastly you can also add no index directive in your robot txt file to pages
or the folders or use wild card all the same as we just discussed
no need to over cure here really but
this might be useful as a second level of security for pages with sensitive information
or do you really really don't want people stumbling across search engines
this is best applied however when your website is new and hasn't be indexed all yet
Feel free to check out our other videos so you can keep fine-tuning and getting the most out of your SEO
and click subscribe to stay updated with new tips and tricks to get you to the
top of the search engine results pages