Yahoo just sent 188 spiders to my forum

Looks like yahoo has come to visit me 188 spiders were on my forums today

 

I was wondering what Yahoo has been smoking lately

 

maybe you should rub out you ip addy for security reasons?

 

Perhaps you should get them to sign-up and keep you company

 

If you're worrying about Yahoo! Slurp or MSNbot being overly aggressive in spidering your site, both will obey the crawler-delay instruction -- add this line to your robots.txt file:

user-agent: *
crawl-delay: 120

Code (markup):

where 120 indicates the number of seconds between page retrievals... set this to whatever you wish (120 is probably a lot longer than necessary - maybe 10 - 30 seconds is more reasonable).

 

Thanks for the great tip

 

another great tip: you could send em to my site if you get hit too hard by em

 

Yeah when Yahoo last deep crawled my site i got hit by atleast 130 different IPs. It was crazy to watch, because my site keeps track of guests as well. And it was just getting bigger and bigger as I was surfin around. Good times, wish yahoo would do it again.

 

Why? Did it do you any good?

There were a lot of weeks a few months back where Yahoo was the busiest spider on the block by far for my sites -- I didn't see a lot of evidence of it doing anything very useful with all that spidering, though.

 

Cant remember if it did any good, i quickly moved to a new domain, currently they only have my main page indexed.