1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

How to Block Yahoo Crawlers in Robots.txt

Discussion in 'robots.txt' started by Bohol, Aug 29, 2008.

0
  1. #1
    Yahoo is indexing our forum site like a drunken, wild bachelor. The site crashes and becomes inaccessible due to a lot of httpd connections made by Yahoo search crawlers.

    How to block Yahoo bots (Inktomi) through robots.txt and .htaccess?
     
    Bohol, Aug 29, 2008 IP
  2. lspence

    lspence Peon

    Messages:
    61
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #2
    User-agent: Slurp
    Disallow: /
     
    lspence, Aug 31, 2008 IP
  3. Bohol

    Bohol Peon

    Messages:
    2,680
    Likes Received:
    75
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Thank you Ispence. Yahoo eats a lot of RAM when its search bot swarm your site. I think Yahoo's slurp needs a major revamp?
     
    Bohol, Sep 1, 2008 IP
  4. V4VoDKa

    V4VoDKa Peon

    Messages:
    33
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Hello My name is Danny , i just noticed this thred and im tryin to post a good reply to and i think that you did really help me with the robots.txt

    thanks a lot:)
     
    V4VoDKa, Sep 2, 2008 IP