User-agent: * Disallow: */cgi-bin/ Disallow: *//forms.lata.net/cgi-bin/multimailer.pl Can someone explain what this code means */ ??? I only know of this code: Disallow: / and not Disallow: */
May be this site has to many subdomains and each further has individual cgi-bin directories. I don't know if it's valid or not but you can test it in google sitemaps account. Login to google sitemaps and find a robots.txt link. Click it. There is a textarea where you can enter a robots.txt code and a INPUT field where you can enter a page url. Paste the code in the textarea and type the file name blocked by robots in the input field. ( http://sub.domain.com/cgi-bin/multimailer.pl) click the verify button below to see the results. If the file above shows as blocked, your robots code is OK, other wise modify it. regards jeet