MSNbot hammering my site this month?

I just checked the stats on one of my sites this morning. So far this month, MSNbot has chewed up 1.38gb of bandwidth. In comparison Googlebot has only had 38m so far.

Sure seems like a lot of bandwidth. The only one close to this is a unknown robot knows as 'spider' at 953mb.

Is this normal?

 

Contact them and ask if there is a problem. I think there are also certain things you can put in your robots.txt file to slow them down a little. I usually welcome all bots to my site though unless they are weird and unknown from some bloke in his basement.

 

The more bandwidth the spiders can chew up the better IMO.

 

I would be worried if the bot wasn't using any bandwidth.

 

True true, at least it has taken interest in your site. Is it costing u anything?

 

I have a site that has been hit over sixty thousand times this month. MSN bots must be having a wild party at that site.

 

Add to robots.txt
Crawl-Delay: 10

 

Can you make a note of WHICH msnbots are hitting your site? For example, msnbot-Media is for Image Search, msnbot-products is for MSN Shopping, and msnbot-news is for News Search. Only the plain msnbot (no hyphen) is for the core search engine.

If some of these don't apply to your site, you can selectively exclude them.

 

Off Topic - Can I ask a question?

How do you know if the search engine spiders have visited your site? I read often about this but I don't know how can I do it.

 

Look at your server logs, they will say Googlebot or Yahoo Slurp or MSN bot.

 

I get a whole load of Unknown Bots... "identified by crawl". Anyhow, bots do eat on my site.

 

Get a good log analyser and check your logs. AWStats is free and gives a good rundown on bot hits from the different SEs.

 

If you're still having problems with our crawler hitting you too much, the formal way to send us a complaint is to complete this form: http://support.msn.com/eform.aspx?p...=support_home_options_form_byemail&ct=eformts

 

it is always good for website if bots visit but even taking huge bandwidth is not good.u can make changes in robo file and stop it.

 

Thanks for the info msndude. I don't really mind the activity of the msnbot hitting my site. I would be more concerned if it stopped hitting the site.