Right at this moment I have 22 yahoo bots crawling my forum, and has been increasing for the past 5 minutes so I don't know when it will reach the max. I couldn't fit them all on my one screenshot. Every bot has a different IP address. So I ask, Just when is "Too Much"? o.o And one more thing, should I be excited?
I had the same thing happen last month to a fricken blog of all things. 3 yahoo ip addresses with hundreds of sub addresses, that were pulling 3g's a piece in bandwidth. I don't know what it was, and there was no response from Yahoo about it.
I get lots of bots from Yahoo on my forum, frequently maxing out database connections. I tried User-agent: Slurp Crawl-delay: 10 Code (markup): in my robots.txt which seemed to help a bit but it's still an invasion Max i got was 180+ at the same time :s Edit: So far, i had 132846 visits from Slurp this month vs. 95890 from Google.
3 yahoo ip addresses with hundreds of sub addresses, that were pulling 3g's a piece in bandwidth. I don't know what it was, and there was no response from Yahoo about it.