I'm a bit mistified at this. I know that the default for googlebot is index, follow but it seems only when I actually use the META tag do my pages get indexed, I don't get it.
I don't specify google itself, I use the deafult for all bots This is what i use: in the robots.txt file: User-agent: * Disallow: in the header of the page: <META NAME="robots" CONTENT="FOLLOW,INDEX"> Cheers John
Hi, thanks, I shall start using robots.txt on all my sites soon, maybe it will benefit me. Thanks for the reply.
If I may ask a question... Do you need to use the robots.txt file or can you use the META tag only? How important is the robots.txt file? just curious, Sassafras
It does not hurt to use these, but there is no benefit in using them. This is exactly what the search engines will understand if the robots.txt and the META are not used. Jean-Luc
The robots.txt file and robots meta tags are used to specify exclusions. If you don't want to exclude robots then don't use them.
<meta name="robots" content="noindex,nofollow"> thats the code if you want to exclude robots from individual pages. i follow it up wit a robots.txt file