Buy Anything On eBay - Hong Kong Hotels - Loans - Consolidation Loans - Loan

PDA

View Full Version : Question about Robots.txt file


hmilesjr
Feb 2nd 2006, 7:57 am
I have the following in a robots.txt file on my site:

User-agent: *
Disallow:
Disallow: /cgi-bin/
Disallow: /mp/

What would cause the search engines to still index the files in these folders? Mainly in Google.

Hubert

gordonfreeman
Feb 2nd 2006, 8:25 am
remove the second line; you don't need it, and it might be the reason