Sem-Advance, what is the point of posting all these links to references on how to construct a good robots.txt file? None of them back up your claim that if you don't have one spiders will leave.
erm, gents we seem to have 2 schools of thought over here. why not we leave it at that and not try to convince one another of the benefits , because it will continue til the cows come home.
Because this isn't a matter of opinion, j3r0m3. This is a matter of Sem-Advance just plain being wrong.
Have to agree. Having or not having a robots.txt file does not make a difference as to whether your site gets indexed or not and saying the robot will leave the site if you do not have one is 100% wrong. If you stipulate parts of your site you do not want robots to index then that's about as good as you can hope for and only some of them will obey.
I get several hits to robots.txt daily, which doesn't exist on any of my sites. So it shows a 404 hit on my stats page, oh well. But I still have Google, MSN and Yahoo crawl HUNDREDS of pages on each site daily. I don't see any point in spending the time creating a robots.txt file when I already get crawled thoroughly.
The point is, you don't need one to get crawled. It's only if you don't want some pages and/or directories indexed. Either in general or by a particular SE. For example, some people exclude Google's Image Bot as it's often unbeneficial (people look at the images and not the pages) and a waste of bandwidth. On the other hand, sitemaps might help in getting indexed by google, assuming the sitemap is submitted to google sitemaps and is compatible with google. Maybe this is what Sem-Advance is confused about. However, a sitemap is by no means essential.