Watching my access logs, it's evident that msnbot does not treat 301s the same as other search engines. I'm not sure if this is new behaviour, nor do I believe this is an indication of "getting things right" (as evidence, Yahoo's piss-poor 301 handling). Upon receiving a 301: - Googlebot and Slurp both immediately return for the new location URL - MSNbot does not follow the redirect, nor does it return for the new location URL in any predictable manner Something interesting to note: - MSNbot-media does return for the new location URL, although it is never immediately.
Actually, the thread you linked to refers to a separate issue. I'm not talking about cached headers or server responses, I'm discussing the behavior of msnbot with regards to following redirects at the moment it encounters one. Yahoo Slurp and Googlebot both behave in a hungry manner, immediately returning to the website and requesting the new location URL provided with the 301. MSNbot, on the other hand, doesn't come back after being served a 301. It may store the location URL and record it for future crawls, but for all intents and purposes it's a "dead end" for that visit. That's a really strange way to configure a crawler. When a crawler visits your site, if no record is made as to the complete status of a header response and its subsequent location responses, there would be no way to get a clear picture of temporary outages, accidental misconfiguration, or any temporary change that shouldn't affect a site's index or permanent ranking. This is the sort of telltale behavior flaw that indicates a much larger philosophical failure, which may explain why MSN is such a horrible search engine.