Blocking of web spider and bots

Discussion in 'Google' started by chavdar, Mar 8, 2010.

0
  1. #1
    Hello,
    At the moment I think my site is overloaded with various bots.
    How to block all web spider and bots other than Google bot, Yahoo, Bing?
    And my second question is how to remove from the Google index a subdirectory that is already indexed?
     
    chavdar, Mar 8, 2010 IP
  2. unknownpray

    unknownpray Active Member

    Messages:
    3,831
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    70
    #2
    You can do this by through google webmaster tools and generating robot file
     
    unknownpray, Mar 10, 2010 IP
  3. berg101

    berg101 Peon

    Messages:
    327
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Yes!! Using Robot.txt and mentioning there Dofollow and Nofollow links ll do ...
     
    berg101, Mar 11, 2010 IP
  4. momofone

    momofone Peon

    Messages:
    24
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You can also use your .htaccess to ban bad bots that ignore robots.txt. I have this in my .htaccess:

    BrowserMatchNoCase Ripper bad_bot
    Order Deny,Allow
    Deny from env=bad_bot

    You can also use this in your robots.txt:

    User-agent: *
    Crawl-Delay: 10

    (substitute the number for what works best for you)
     
    momofone, Mar 11, 2010 IP
  5. webcosmo

    webcosmo Notable Member

    Messages:
    5,840
    Likes Received:
    153
    Best Answers:
    2
    Trophy Points:
    255
    #5
    bots might not follow the instructions from the robots.txt file.

    your best best is coding to check the uger agent form the http referrer. if its not a visitor or certain bots allowed, redirect to somewhere.
     
    webcosmo, Mar 11, 2010 IP