Futher questions on "robots.txt" file

This thread is a some kind of continuing of the I just applied adsense.. one ...


I am curious: why not to use "robots.txt" (especially if they allow Google to index) ??


For instance my "robots.txt" (I was playing with them long time ago though) allows all bots to index, but just restricts access to some "sample" directory (used for testing), so this doesn't have any impact on the actual site's documents indexing. And what about "robots.txt" vs. robots meta tag (i.e. if they are conflicting)


P.S. -- Also, I am not totally sure, does the "robots" principle apply also to sub-domains (or separate servers on these domains); like for instance to my http://tadej-ivan.50webs.com/ site variant, or they aplly only to the domains (i.e. to 50webs.com) ??


thanks, tayiper

 

Hi tayiper,

First of all, Google will index your site, if there is no robots.txt. You can only use robots.txt to disallow access to some parts of your site.

robots.txt contains instructions addressed to web robots. META tags are commands included in web pages; some of them are addressed to search engines.

You asked what happens if they are conflicting ? If robots.txt disallows access to some web pages, a compliant robot will not read the disallowed pages, thus its search engine will not be aware of the contents of these pages (it will not know which META tags they include).

Jean-Luc

 

you will need to place a robots.txt file for each subdomain also: robots.txt

 

Exactly! For tadej-ivan.50webs.com, robots.txt should be located at tadej-ivan.50webs.com/robots.txt .

Jean-Luc

 

Thanks all for the replies ...


tayiper