When i has similar problems with yahoo it turned out the Crawl-delay was been implemented per bot and from looking at my server logs it looks like msn bot is the same way. Fine when they use only one bot but not much good when they use 20+ at once. You may find you need to increase the Crawl-delay by the number of bots to get the desired effect.
Yeah. You are right. Their crawlers are badly designed. The worst part is that they do not follow instructions mentioned in robots.txt or meta-robots tag. It simply disobeys those instructions and MS says that they know about this bug and are trying to fix it ASAP. Look here ---> http://www.provenseo.com/2009/08/bing-indexing-noindex-nofollow-content/
Wow... that would be completely moronic if that's the case. So they want to use multiple computers to act as a single process (a search engine spider), yet they haven't figured out how to get their own machines to communicate with each other yet. {rolls eyes} How exactly do they think they are going to make a better search engine than Google? Throwing money at advertising doesn't seem to be fixing any of their core problems so far. Maybe a few billion more in advertising will make the spider work right. Then maybe they can brainstorm about how to make the web pages they have indexed searchable by relevancy. A novel idea.
I totally agree it's totally moronic, and should not happen. However, when i last looked at latest visitors page in cpanel to see what they were doing.... where they group the pages viewed by each IP ... i could see msnbot (also yahobot) obeying crawl delay within each bot .... but with 20 bots on at once, a 2 second delay means as a whole they were visiting 10 pages per second as a group. Been on shared hosting, I ended up putting a 60 second Crawl-delay on one of my forums It may well be a bug that they will sort out, but it should be easy enough to see/test if that is what is also happening here. vagrant
Yes we have to ignore msn bot or may be msn bot come due to bing is launched and the bing is latest technology