msnbot

Anybody know why msn would be sucking up tons of bandwidth off a blog?

For some reason in Aug msnbot crawled 2.5 gigs of data on my blog, the previous months it only did about 500 megs.

 

do you have lot of images as well on your blogs ?

 

I saw about a ten fold increase in MSN crawling last month. It dwarfed all other SE's, but the bandwidth wasn't nearly as bad as yours.

 

Use your stats to see what files are wasting the most bandwidth.

If it is text (html, php, asp, ...) try using http compression (gzip), it saves huge ammounts of bandwith... a 50 kb page easely turns into 5 KB, even less.
What i'm not sure is if msnbot supports it, i'll check it and let you know.

If it is the images, you can also block bots from getting to your images with robots.txt

 

There's maybe a dozen small images on the blog, so it's definately not that.

I'm not really worried about bandwidth usage right now (I have 75GB/month allocated), but it's just very strange for the bot to be doing it. It's only crawling the html pages mostly, but it's hitting all of them like 500+ times/month.

 

Basically, that's just msnbot for ya!

Lots of people actually block it because it gobbles so much (not smart- IMO). If it's a problem, you can contact them and tell them to slow down a little.

 

I wouldn't block the bot unless I didn't care about the five extra visitors they send to my site each day.