How to Block Yahoo Crawlers in Robots.txt

Discussion in 'robots.txt' started by Bohol, Aug 29, 2008.

0
  1. #1
    Yahoo is indexing our forum site like a drunken, wild bachelor. The site crashes and becomes inaccessible due to a lot of httpd connections made by Yahoo search crawlers.

    How to block Yahoo bots (Inktomi) through robots.txt and .htaccess?
     
    Bohol, Aug 29, 2008 IP
  2. lspence

    lspence Peon

    Messages:
    61
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User-agent: Slurp
    Disallow: /
     
    lspence, Aug 31, 2008 IP
  3. Bohol

    Bohol Peon

    Messages:
    2,680
    Likes Received:
    75
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thank you Ispence. Yahoo eats a lot of RAM when its search bot swarm your site. I think Yahoo's slurp needs a major revamp?
     
    Bohol, Sep 1, 2008 IP
  4. V4VoDKa

    V4VoDKa Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hello My name is Danny , i just noticed this thred and im tryin to post a good reply to and i think that you did really help me with the robots.txt

    thanks a lot:)
     
    V4VoDKa, Sep 2, 2008 IP