Can robot.txt be accessed by public?

Discussion in 'robots.txt' started by alvin1972, Mar 16, 2010.

0
  1. #1
    How to prevent robot.txt being accessed by public? Any idea how to implement this? Please comments. Thanks
     
    alvin1972, Mar 16, 2010 IP
  2. shallowink

    shallowink Well-Known Member

    Messages:
    1,218
    Likes Received:
    64
    Best Answers:
    2
    Trophy Points:
    150
    #2
    yes it can. why would you want to prevent access? isn't like its a top secret list or anything. databases exist for almost all the search bots.
     
    shallowink, Mar 16, 2010 IP
  3. bigcat1967

    bigcat1967 Active Member

    Messages:
    1,333
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    98
    #3
    tell you what - you may not need it if you want all bots to visit your site...
     
    bigcat1967, Mar 17, 2010 IP
  4. tasteemarketing

    tasteemarketing Peon

    Messages:
    15
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I don't know why you would want to block it, there is nothing secret, or special about the robot.txt file? I also don't think the public will see your robot.txt file unless they get into your server.
     
    tasteemarketing, Mar 23, 2010 IP
  5. fierceservers

    fierceservers Peon

    Messages:
    338
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #5
    As with the other answers here, of course it's public. How do you think search engines can read it to begin with? ;)
     
    fierceservers, Mar 23, 2010 IP