helloo there every1 i m bit newbie in dis.. Can any1 tell me if i have insterted proper robots.txt on my site www.buyrimonabant.co.uk/robots.txt and is it necessary to insert robots command in all pages including index?? Plz Comment Thanx in advance
you dont need robots.txt unless you want to prevent something from being indexed your robots.txt should be the same as a 404 return code..since it has nothing to disallow
as far as the robots meta tags... the biggest waste of space as far as code on a page is to use the robots meta to allow spiders (follow, index, all) This is the natural assumed state for a spider. The robots exclusion standard is just that... an exclusion. I wouldn't include the robots meta if all you want is to tell spiders that it's ok to visit your page or follow the links on it. It just hurts your code:text ratio.
MSN still recommends to use robot.txt. Using robot.txt does not hurt you always, in fact it is pretty useful sometimes. It prevents useless bots from visiting your site and it instructs the bots, and various cralers the time frame to revisit your site, etc Here is a nice link to read.