Possible loophole with Robots.txt?

Discussion in 'robots.txt' started by pokerface, Mar 25, 2009.

0
  1. #1
    Hi,
    I was wondering what if someone access mysite.com/robots.txt and find out the pages which they shouldn't see it or pages which I want to hide. How do I make sure people cannot see my robots.txt or are there any possible way to go around this?

    I have tried on a few clickbank sites and when i access their robots.txt, it auto. directs me to their index page.

    how do i do that?

    thanks:D
     
    pokerface, Mar 25, 2009 IP
  2. jik34

    jik34 Active Member

    Messages:
    586
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    83
    Digital Goods:
    4
    Articles:
    2
    #2
    In .htaccess add the code

    
RewriteCond {HTTP_USER_AGENT} ^Mozilla 
RewriteCond %{HTTP_USER_AGENT}!(Slurp¦surfsafely) 
RewriteRule ^robots\.txt$ /someotherfile [L]
    Code (markup):
    "someotherfile" could be blank, or a it could be a fake robots.txt.

    source: http://www.webmasterworld.com/forum93/74.htm
     
    jik34, Mar 26, 2009 IP
  3. linkdealer

    linkdealer Active Member

    Messages:
    138
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    90
    #3
    thanks jik 34 for this nice information
     
    linkdealer, Jun 9, 2009 IP