Ask Jeeves is killing me

I have just checked my stats for one of my sites and Ask Jeeves spider has spidered my site nearly 7000 times this month useing just under 1GB of bandwidth. Has anyone else seen this type of activity? also what is ask jeeves now as they do not appear to have a search facility so who is it really spidering my site?

 

I was wondering the same thing. All of a sudden Ask Jeeves is going crazy on my sites. It's out of control. I don't think they send me any traffic so it's weird.

 

I wonder if there is a way to tell their spider to slow down a bit.

 

I've never used it but I know that MSN allows crawl delay through Robots file, I would suspect that others do as well.

 

I just block it. It brought me like 1 or 2 visitors in half a year but ate a lot of bandwidth.

 

For me AJ is growing and is actually ranked third in Search Engine referrals. Maybe it has something to do with content.

 

Just block it, that's what I would do in that situation.

 

Think i'm the top victim though
AJ ate 160,000 pages - (6,4GB) on August. There was not any of my pages in it's index next month after it stopped indexing. Then 94,000 pages become searchable on AJ - it was brinning up to 40 visitors daily. But in 3 days the crawler got down to work again. This time it "stripped" me with 240,000 (10,2Gb) in 5 days - and then my site disappeared from index again. It lasts 1,5 months. Think i will bann AJ in robots.txt

 

So, how do you stop a crawler from visiting your site altogether?

 

"So, how do you stop a crawler from visiting your site altogether?"
robot.txt

 

Could you please tell me what I should type into robots.txt?

Thanks!

 

They must just not like your sites guys cos on one of mine I got these stats.

AskJeeves - hits 31+39 - bandwidth 508.49 KB - last visit 25 Apr 2006 04:27

Ask Jeeves - pages 340 4.1 % - hits 340 3.4 %


I like um myself 3rd best provider out of search engines for me.

Yahoo is the let down for me Google rocks and MSN we do well too.

Thanx John

 

http://www.robotstxt.org/wc/exclusion-admin.html

 

i have this problem too... in htaccess you dennied this bot

 

1gb of bandwidth isn't very much. The google bot has used up 15gb bandwitdh this month

 

Why are you guys getting so many bot hits? My largest highest traffic site (~500 uniques per day), all bots combined is only 470MB this month

 

Me too, I never see that kind of bots traffic on my site. I suppose if you have a site that has new content everyday that it would invite the bots more often. just a guess...

 

Send them an invoice for the bw usage.

 

My logfile rotated last night at 1:00am and the ask bot has already crawled 8,900 pages today:

grep ask  /etc/httpd/domlogs/sitename.com -c
8893

Code (markup):

This has been going on for three months and I bet I haven't had 10 referrals from them. . . It's not that big a deal but it would be nice if I got SOMETHING in return.