1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Is there a way to make the bots respect the robots.txt file at all times?

Discussion in 'robots.txt' started by qwikad.com, Jan 31, 2012.

0
  1. #1
    I created the robots.txt file placed this code in it:

    -----------------------

    User-agent: *Disallow: /


    User-agent: Googlebot
    Allow: /


    User-agent: Yahoo-slurp
    Allow: /


    User-agent: Msnbot
    Allow: /


    User-agent: Bingbot
    Allow: /

    -----------------------

    Everything is working fine but, if someone decides to avoid my robots.txt file then the whole thing has very little use against malicious bots. What command/file/script can be used to make ALL bots respect the robots.txt file?
     
    qwikad.com, Jan 31, 2012 IP
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    7,151
    Likes Received:
    1,656
    Best Answers:
    29
    Trophy Points:
    475
    #2
    Just found the answer to my own question. Added a bunch (over 200) bad-bot names to my site's directory. Still looking for more...
     
    qwikad.com, Jan 31, 2012 IP
  3. tiffanywilliams12i2

    tiffanywilliams12i2 Peon

    Messages:
    164
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    are you looking to block bad traffic?
     
    tiffanywilliams12i2, Apr 10, 2012 IP