Recently I uploaded robots.txt to a friends website so spiders will not crawl anything located in his CMS control panel. So I created a .txt document that reads... User-agent: * Disallow: /amass I see that in the Google webmaster tools section they have robots.txt generator. When I created the file there it read like this... User-agent: * Disallow: /amass Allow: / Personally I would not place an allow directive in my robots.txt file since the default behavior is to allow. Which is the correct way to write this file? I want every spider to crawl his site and not get confused when they see robots.txt!