Is Robots txt just a simple text file. Can I create it? I don't see any robots.txt in my server and google webmasters tools says that: Analysis of cached robots.txt Last downloaded: August 24, 2007 7:33:38 PM PDT Status: 404 (Not found)
robots.txt is EXACTLY that.... robots.txt You make it... put it in your root folder and call it... you got it! robots.txt NOTE: I recommend everyone have a robots.txt file in all their sites.
You don't need one unless you need to block a bot from crawling a specific area of your site. No robots.txt means crawl it all.
Actually, you should have one, and it should be required (even though it's not). Even if it's blank (or set to allow everything to crawl), it'll still benefit your site. Not from an SEO perspective, but it'll help prevent all those false positives in your server logs (just like using a favicon.ico file will help clean up those logs as well). Think about it. Would you rather skip past every "error" for a missing robots.txt or favicon.ico file in your server logs or would you rather tell right away which links are broken and what needs to be fixed? I'd choose the latter.