When I check the traffic stats in my domain, I keep seeing this /robots.txt Http Code: 404 Date: Apr 04 12:49:14 Http Version: HTTP/1.0 Size in Bytes: - Referer: - Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) What is this? I have checked my domain and there is no such file as robot.txt. Do I need one there? If yes then what should it contain?
Usually the "good" seach engines will read the robots.txt and see what you do not want searched. The "bad" ones won't though. And you should have the robots.txt in there with a link to your XML sitemap since most search engines are using this method to find your XML sitemap. You just add something like Sitemap: http://www.exmaple.com/sitemap.xml Code (markup):
Whenever bot come to your website it first look for robots.txt to check whether you have prohibited any SE Bot or private pages to index..if there is not any robots.txt in your website, it simply means you don't want to restrict any of the SE Bot or any private pages...
I believe you'll find this useful: http://rauru.com/2008/04/why-you-should-have-a-robotstxt-in-your-sites/ Hope that helps
Some people say if your not interested in excluding certain areas of your site, it doesn't matter if you have one at all.... But I always like to put one anyway, just so that it doesn't draw a 404 log hit and it gives the se's confidence that it is there and your interested in using it to communicate.
Search engine crawler first try to access Robots.txt to ascertain webmaster instruction given to crawler. The log file writes the crawler access wither it is on server or not