.htaccess instead of robots.txt?

Discussion in 'Apache' started by zver0, Feb 17, 2006.

0
  1. #1
    Can .htaccess be used instead of robots.txt to prevent bots from crawling/indexing certain directories/files?
     
    zver0, Feb 17, 2006 IP
  2. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #2
    .htaccess will ban them. Robots is just a suggestion and you can only hope they obey your file.
     
    Nintendo, Feb 17, 2006 IP
  3. zver0

    zver0 Peon

    Messages:
    35
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Can this be done without cloaking? I mean is there a sure way to ban SE bots, without IP delivery?
     
    zver0, Mar 2, 2006 IP
  4. Nintendo

    Nintendo ♬ King of da Wackos ♬

    Messages:
    12,890
    Likes Received:
    1,064
    Best Answers:
    0
    Trophy Points:
    430
    #4
    Something like

    <Limit GET POST>
    order allow,deny
    allow from all
    deny from 12.154.217
    </Limit>

    can ban them. I'm guessing IP is the only way to ban them, unless you can ban them using there name, like Googlebot.
     
    Nintendo, Mar 2, 2006 IP