over 500 spiders from Yahoo on a daily basis

[Devilfish]

I have a forum and I get 500+ (sometimes 800+) Yahoo spiders daily. Why is there so many? I only get 1 or 2 Google spiders.

I guess that's the difference is the algorithms and the technology they use.

[SticKer]

i don't know but for the past 3-4 days yahoo bots are acting weird. I see 50-60 bots all over my site at a given time and then they are all gone again after sometime they come back. Today it seems to be normal.

[jpl691]

yeh the google bot i think lives at my forum

[leo7068]

you can limit the spiders with robots.txt right?

[Devilfish]

I know you can limit the bot visits but my forum is new(ish) so I need all the visibility I can get lol.

Just wondered, that's all.

[Gnome]

Yeah i noticed lots of yahoo bots around.
And google lives at my forum as well

[iceman2000]

I am new to this.

How do you determine that bots are on your site and how do you limit their visits?

I am assuming that everytime they visit, they chew up some bandwidth.

[snappingpig]

I get the same thing, Yahoo just seems crappy for some reason because I'm never on there. Google is the best though and they can get the job done every day without 100s of spiders.

[soccerfriend]

same here I had 1000 ++ yahoo spiders.. maybe yahoo improving their search engine

[Neticule]

If its a forum would it say there are 1000+ guests viewing? lol

[cheaptraffic]

If too many bots visits your site, would it make your site slower?

[high_flying]

No not really. Especially if the images and html files are small

[obay]

i've had this for the past couple of days... and I'm quite happy with the result now...

http://search.yahoo.com/search?p=mil...8&fp_ip=UK&vc=

In the result, it's ranked hyper with Yahoo then Google.

[rsfrs]

I think better u control from robot.txt

[sachin410]

same with my forum.

Yahoo bots are on steroids.

Originally Posted by rsfrs

I think better u control from robot.txt

Originally Posted by leo7068

you can limit the spiders with robots.txt right?

can someone please explain how?

[djixas]

Originally Posted by Devilfish

I have a forum and I get 500+ (sometimes 800+) Yahoo spiders daily. Why is there so many? I only get 1 or 2 Google spiders.

I guess that's the difference is the algorithms and the technology they use.

I got hundreds of connections at same time from Google spiders which crashed my old server once (when Yahoo did the same).

[AdenFraser]

Getting 800 - 1000 Yahoo per day on my forum. Sometimes 50 + at one moment, yet only had 6 Google spiders visit in the past 24hours.

Wierd.

[joeschmo12]

yeah yahoo spiders.. aka yahoo slurp spiders... they just never seem to leave your forums.. whereas google sends a few spiders a day and can get the same amount of info. Shouldnt that send a message to yahoo about fixing their slurps?

[sydneyaus]

The past few days Yahoo has crawled my site daily 70-80 times 1 page at the time. Google just comes by 5 times a day and takes 25-40 pages at a time. strange

[Jim4767]

I'm using StatCounter. Can anyone who knows StatCounter point me to the portion of StatCounter where I can check on the frequency of the spiders' crawling activity? Thanks.