Is this a good robots.txt? #User-agent: lycra #Disallow: / #User-agent: * #Disallow: /tmp #Disallow: /logs User-agent: * Disallow: What does the User-agent: * Disallow: tell the spiders? That all are allowed or none? If you haven't figured it out by now, I am working with another company to help them. Just going over their stuff. I always thought the above should read User-agent: * Disallow: / To allow all others....
User-agent: * Disallow: / = no access to all robots User-agent: * Disallow: gives access to all robots see ww w.robotstxt.org/wc/norobots.html for details