Ask.com's bot

Discussion in 'Site & Server Administration' started by chachi, Jan 13, 2005.

  1. #1
    It seems that ask.com's bot is in love with two of our sites. The past two day have been especially silly. One site received 11k visits yesterday and today one is just off the 13k mark. I have no idea what they are doing with all of that info as we rarely receive any traffic from them. Funny thing is, these are "newer" sites.
     
    chachi, Jan 13, 2005 IP
  2. pwaring

    pwaring Well-Known Member

    Messages:
    846
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    135
    #2
    Have you emailed them to ask if there's a reason why they're crawling you so much? It might be that you've got a loop in your site that they're following or a bug in their crawler. Either way they'd probably appreciate knowing about the problem.
     
    pwaring, Jan 13, 2005 IP
  3. chachi

    chachi The other Jason

    Messages:
    1,600
    Likes Received:
    57
    Best Answers:
    0
    Trophy Points:
    0
    #3
    this is across multiple sites. The pages they are requesting are all different. Not sure there is anything "wrong" other than the fact that they are grabbing an insane number of pages
     
    chachi, Jan 13, 2005 IP
  4. pwaring

    pwaring Well-Known Member

    Messages:
    846
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    135
    #4
    Well, if it's a well-behaved bot it shouldn't be grabbing an insane number of pages, unless it's leaving a reasonable gap (say two seconds) between each request. :)
     
    pwaring, Jan 13, 2005 IP
  5. chachi

    chachi The other Jason

    Messages:
    1,600
    Likes Received:
    57
    Best Answers:
    0
    Trophy Points:
    0
    #5
    heh, yeah. The grab one about every 3 to 4 seconds when it goes on a run
     
    chachi, Jan 13, 2005 IP
  6. Refrozen

    Refrozen Peon

    Messages:
    318
    Likes Received:
    9
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Wow. I'd send them an email or look for support forums run by them... e-mail is the best choice.
     
    Refrozen, Jan 13, 2005 IP