In the last few days our website literally was flooded with thousands and thousands of page views by MSNBot! This crazy little bot indexes the home page every few seconds though it never changed in the last few days! All the other spiders crawl the site as they should. MSNBot is the only one indexing like crazy. At MSN Live Search Help I read, that the little biest can actually be tamed by setting the crawl-delay in the robots.txt. Like: ...which sets the bot's crawl speed to a single page every 180 seconds. Did anyone ever try that? Does it work? What else can we do in order to prevent MSNBot from running amok?
Yes, crawl-delay works ... googlebot of course, ignores it. Why should google conform, right? 180 seems a bit high, try 30 or 60 and see how that works.
MSNBot will require a lot of resources, its unlikely their admins would allow such a load on their own systems. Did you verify the IP address? User agent means nothing, it can be (and being) faked.
No, I didn't verify the IPs. Looks like msnbot follows the crawl-delay rule. Now we have a new customer: the baiduspider... *sigh