Question about Robots.txt file

Discussion in 'robots.txt' started by hmilesjr, Feb 2, 2006.

0
  1. #1
    I have the following in a robots.txt file on my site:

    User-agent: *
    Disallow:
    Disallow: /cgi-bin/
    Disallow: /mp/

    What would cause the search engines to still index the files in these folders? Mainly in Google.

    Hubert
     
    hmilesjr, Feb 2, 2006 IP
  2. gordonfreeman

    gordonfreeman Peon

    Messages:
    169
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    remove the second line; you don't need it, and it might be the reason
     
    gordonfreeman, Feb 2, 2006 IP