Ask Jeeves Killing my site

Discussion in 'Site & Server Administration' started by jfontestad, Jan 30, 2006.

0
  1. #1
    At about mid-January AskJeeves started to JeeveBomb one of my sites. As of today it has crawled 273k+ pages for a total of 5GB+. Should I disallow this bot in my robots.txt? Or will this be beneficial for me in the future?
     
    jfontestad, Jan 30, 2006 IP
  2. Dekker

    Dekker Peon

    Messages:
    4,185
    Likes Received:
    287
    Best Answers:
    0
    Trophy Points:
    0
    #2
    meh, askjeeves is one of the dead search engines if you ask me.
     
    Dekker, Jan 30, 2006 IP
  3. av1

    av1 Active Member

    Messages:
    168
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    60
    #3
    add this to ur robot.txt:

    User-agent: Teoma
    Crawl-delay: 1000
     
    av1, Jan 31, 2006 IP
  4. markhutch

    markhutch Peon

    Messages:
    357
    Likes Received:
    22
    Best Answers:
    0
    Trophy Points:
    0
    #4
    From my experience, once they are finished you won't see them again for a long time. I'd just hang in there and hope for the best.
     
    markhutch, Jan 31, 2006 IP
  5. falco85

    falco85 Peon

    Messages:
    721
    Likes Received:
    40
    Best Answers:
    0
    Trophy Points:
    0
    #5
    I'd love to add this to my robots.txt

    User-agent: anything_except_googlebot
    Disallow: \

    User-agent: Googlebot
    Imploration: Please, please please crawl me I'll do anything to get crawled and fully indexed :D
     
    falco85, Jan 31, 2006 IP
    Dio likes this.
  6. sarathy

    sarathy Peon

    Messages:
    1,613
    Likes Received:
    76
    Best Answers:
    0
    Trophy Points:
    0
    #6
    lol @ faloc.,

    I still get a very few visits from ask., Something is better than nothing ;)
     
    sarathy, Jan 31, 2006 IP
  7. Cyclops

    Cyclops sensei

    Messages:
    1,241
    Likes Received:
    72
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Ask Jeeves crawled two of my sites using up all their bandwidth three months in a row....Was it worth it, I think so as the traffic has outnumbered the other search engines since then.
    I haven't bothered to check the serps but my terms must be ranking at the top.
     
    Cyclops, Jan 31, 2006 IP
  8. guest

    guest Well-Known Member

    Messages:
    1,068
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    155
    #8
    similar incident happening with two of my sites,

    google 390k+412 crawlings
    yahoo 27k+
    consuming hell of bandwidth
     
    guest, Jan 31, 2006 IP
  9. Serious

    Serious Peon

    Messages:
    185
    Likes Received:
    6
    Best Answers:
    0
    Trophy Points:
    0
    #9
    bandwidth is cheap, cpu is not.
    Heavy PHP pages are a damn.
     
    Serious, Jan 31, 2006 IP
  10. livingearth

    livingearth Well-Known Member

    Messages:
    1,469
    Likes Received:
    83
    Best Answers:
    0
    Trophy Points:
    140
    #10
    I would think that being spidered by the search engines would be beneficial rather than a nusiance. Bandwidth is pretty cheap actually. How much bandwidth are we talking about? Need a new host?
     
    livingearth, Jan 31, 2006 IP
  11. tdave

    tdave Peon

    Messages:
    33
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #11
    that a lot ,mate..
     
    tdave, Mar 11, 2009 IP
  12. AquariusRyan

    AquariusRyan Peon

    Messages:
    109
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Way to bring up a dead/old topic....

    Plus bandwidth is cheaper now so that's really even less then it was before ;)
     
    AquariusRyan, Mar 12, 2009 IP