MSNBot being too aggressive?

[digitalpoint]

So I was trying to figure out where a huge influx of used bandwidth was coming from for this site, and it *looked* like it was msnbot from the brief look at the logs. I ended up blocked a giant allocation of Microsoft IP addresses (65.52.0.0 - 65.55.255.255, which is 262,142 IP addresses) as a quick test to see if my bandwidth usage went back to normal...

/sbin/route add -net 65.52.0.0 netmask 255.252.0.0  reject

Sure enough, sustained bandwidth output went down by about 8Mbit/sec... So msnbot alone was eating up 8Mbit 24/7.

I ended unblocking the IP addresses and adding a 2 second delay for each page msnbot can get via robots.txt (Crawl-delay: 2).

Anyone else seeing this sort of stupidity from msnbot (149 robot requests every 10 seconds)?

Check out a 10 second sample of hits from msnbot. Dumb.

Originally Posted by 10 Second Sample

65.55.106.121 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-972423.html HTTP/1.1" 200 1455 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-942477.html HTTP/1.1" 200 1484 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=1191222&goto=newpost HTTP/1.1" 302 26 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.209.53 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=253219 HTTP/1.1" 200 14779 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=755269 HTTP/1.0" 200 61132 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:01 -0700] "GET /archive/index.php/t-1225471.html HTTP/1.0" 200 8224 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-813390.html HTTP/1.1" 200 1584 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-597159.html HTTP/1.1" 200 2254 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-517908.html HTTP/1.0" 200 3781 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-1095264.html HTTP/1.1" 200 2473 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-921100.html HTTP/1.0" 200 3695 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:03 -0700] "GET /forumdisplay.php?f=13&page=11 HTTP/1.0" 200 85489 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-851623.html HTTP/1.1" 200 1431 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-352881.html HTTP/1.1" 200 1400 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=70912 HTTP/1.0" 200 67205 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:04 -0700] "GET /archive/index.php/t-732510.html HTTP/1.1" 200 1432 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:04 -0700] "GET /archive/index.php/t-392350.html HTTP/1.0" 200 6540 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:05 -0700] "GET /archive/index.php/t-88196.html HTTP/1.0" 200 4879 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=1171743 HTTP/1.0" 200 120466 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=630833 HTTP/1.1" 200 10736 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:06 -0700] "GET /archive/index.php/t-362095.html HTTP/1.1" 200 1938 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?t=674618&page=9 HTTP/1.1" 200 14632 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?t=836179 HTTP/1.1" 200 7891 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:07 -0700] "GET /showthread.php?t=934232 HTTP/1.1" 200 9911 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:07 -0700] "GET /showthread.php?t=83494 HTTP/1.0" 200 39651 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:07 -0700] "GET /archive/index.php/t-513031.html HTTP/1.0" 200 5025 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:08 -0700] "GET /archive/index.php/t-96193.html HTTP/1.0" 200 3707 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:08 -0700] "GET /showthread.php?p=9666008 HTTP/1.0" 200 95145 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:08 -0700] "GET /archive/index.php/t-881916.html HTTP/1.1" 200 1561 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-587867.html HTTP/1.0" 200 16434 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.209.53 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=37573 HTTP/1.1" 200 13904 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:01 -0700] "GET /archive/index.php/t-818464.html HTTP/1.1" 200 2488 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=1284751 HTTP/1.0" 200 57238 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=1146469 HTTP/1.0" 200 139035 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=595506 HTTP/1.1" 200 12450 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-170192.html HTTP/1.1" 200 2055 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-276105.html HTTP/1.0" 200 3745 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=1191222#post10685133 HTTP/1.1" 200 11673 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-1063956.html HTTP/1.0" 200 3225 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-630820.html HTTP/1.0" 200 10295 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-1014676.html HTTP/1.1" 200 1436 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=597876 HTTP/1.1" 200 10671 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=76281 HTTP/1.0" 200 59246 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=1234416 HTTP/1.1" 200 8082 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-1089076.html HTTP/1.0" 200 138454 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:04 -0700] "GET /archive/index.php/t-141872.html HTTP/1.0" 200 6793 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=198040 HTTP/1.1" 200 9681 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:05 -0700] "GET /member.php?u=28519 HTTP/1.1" 200 8199 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=50392 HTTP/1.1" 200 11221 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=137040 HTTP/1.1" 200 10252 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=890534 HTTP/1.0" 200 59386 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:06 -0700] "GET /archive/index.php/t-655418.html HTTP/1.1" 200 4203 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:07 -0700] "GET /archive/index.php/t-139369.html HTTP/1.0" 200 20402 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:07 -0700] "GET /archive/index.php/t-975418.html HTTP/1.1" 200 1269 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:07 -0700] "GET /showthread.php?t=208217 HTTP/1.0" 200 98837 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:09 -0700] "GET /archive/index.php/t-839622.html HTTP/1.1" 200 1245 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:09 -0700] "GET /showthread.php?t=853000 HTTP/1.1" 200 10673 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=247345 HTTP/1.1" 200 11675 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-590385.html HTTP/1.0" 200 5810 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=696054 HTTP/1.0" 200 42023 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=134232 HTTP/1.0" 200 44411 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=167893 HTTP/1.1" 200 7841 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:01 -0700] "GET /archive/index.php/t-947476.html HTTP/1.0" 200 13213 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=199348 HTTP/1.1" 200 9842 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:01 -0700] "GET /archive/index.php/t-16059.html HTTP/1.0" 200 4716 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:01 -0700] "GET /archive/index.php/t-757369.html HTTP/1.1" 200 1606 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=411935 HTTP/1.1" 200 13761 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=1191222&goto=lastpost HTTP/1.1" 302 26 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-923541.html HTTP/1.0" 200 8609 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.209.53 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=505451 HTTP/1.1" 200 8159 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:02 -0700] "GET /itrader.php?u=138593 HTTP/1.1" 200 5296 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=18122 HTTP/1.0" 200 51703 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=17831&page=2 HTTP/1.1" 200 10736 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=524942&page=3 HTTP/1.1" 200 13267 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-1169365.html HTTP/1.0" 200 3780 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=207463 HTTP/1.0" 200 76897 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-537465.html HTTP/1.1" 200 1695 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-355095.html HTTP/1.1" 200 1853 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.209.53 - - [31/Jul/2009:12:13:03 -0700] "GET /member.php?u=10672 HTTP/1.1" 200 7885 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-700796.html HTTP/1.1" 200 5422 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:03 -0700] "GET /archive/index.php/t-885247.html HTTP/1.0" 200 3689 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:04 -0700] "GET /archive/index.php/t-1192332.html HTTP/1.1" 200 2618 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:04 -0700] "GET /itrader.php?u=86788 HTTP/1.1" 200 5662 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:04 -0700] "GET /archive/index.php/t-86347.html HTTP/1.0" 200 7341 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?p=5359715#post5359715 HTTP/1.1" 200 15167 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=1292906 HTTP/1.1" 200 8392 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=314574 HTTP/1.1" 200 12623 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:04 -0700] "GET /archive/index.php/t-869681.html HTTP/1.1" 200 1944 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=1229174 HTTP/1.0" 200 41036 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=1206670 HTTP/1.0" 200 121752 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:05 -0700] "GET /itrader.php?u=101708 HTTP/1.1" 200 6599 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:05 -0700] "GET /archive/index.php/t-1091042.html HTTP/1.0" 200 10757 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=46453 HTTP/1.0" 200 77576 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:05 -0700] "GET /archive/index.php/t-1185265.html HTTP/1.0" 200 2946 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:06 -0700] "GET /archive/index.php/t-899850.html HTTP/1.0" 200 30068 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?t=1240318 HTTP/1.1" 200 12986 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:07 -0700] "GET /itrader.php?u=40839 HTTP/1.0" 200 49097 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.107 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?t=422458 HTTP/1.0" 200 110756 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:07 -0700] "GET /showthread.php?t=1233343&page=9 HTTP/1.1" 200 15406 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?t=170918 HTTP/1.0" 200 128344 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.107 - - [31/Jul/2009:12:13:07 -0700] "GET /showthread.php?t=633978 HTTP/1.0" 200 63496 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:07 -0700] "GET /showthread.php?t=767230 HTTP/1.0" 200 46835 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:08 -0700] "GET /archive/index.php/t-278066.html HTTP/1.1" 200 1613 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:08 -0700] "GET /archive/index.php/t-216125.html HTTP/1.1" 200 1363 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:08 -0700] "GET /showthread.php?t=23906 HTTP/1.1" 200 12185 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:08 -0700] "GET /showthread.php?t=194264 HTTP/1.1" 200 10373 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.107 - - [31/Jul/2009:12:13:08 -0700] "GET /showthread.php?t=1260506 HTTP/1.0" 200 55633 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:08 -0700] "GET /showthread.php?t=116825 HTTP/1.1" 200 10560 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:09 -0700] "GET /showthread.php?t=804979 HTTP/1.1" 200 12113 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:09 -0700] "GET /showthread.php?t=513914 HTTP/1.1" 200 8739 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.107 - - [31/Jul/2009:12:13:09 -0700] "GET /showthread.php?t=890967 HTTP/1.0" 200 128953 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-655857.html HTTP/1.1" 200 2046 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-222299.html HTTP/1.0" 200 5105 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=1411842&page=2 HTTP/1.0" 200 40598 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-370886.html HTTP/1.1" 200 3602 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-755779.html HTTP/1.1" 200 6850 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:00 -0700] "GET /showthread.php?t=278038 HTTP/1.0" 200 106343 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:00 -0700] "GET /archive/index.php/t-611268.html HTTP/1.1" 200 1612 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:01 -0700] "GET /archive/index.php/t-594398.html HTTP/1.1" 200 1333 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.117 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=1084587 HTTP/1.1" 200 15791 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=47715&page=4 HTTP/1.0" 200 51682 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=328982 HTTP/1.1" 200 8941 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:01 -0700] "GET /showthread.php?t=619725 HTTP/1.0" 200 54580 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=526564 HTTP/1.0" 200 49558 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.118 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-830159.html HTTP/1.1" 200 1305 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-621797.html HTTP/1.0" 200 3962 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:02 -0700] "GET /archive/index.php/t-38425.html HTTP/1.1" 200 1310 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.209.53 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=71731 HTTP/1.1" 200 13108 "-" "msnbot/1.1 (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=38969 HTTP/1.0" 200 60690 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.244 - - [31/Jul/2009:12:13:02 -0700] "GET /showthread.php?t=401996 HTTP/1.0" 200 132011 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=762988 HTTP/1.1" 200 10822 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=457543 HTTP/1.0" 200 91243 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:03 -0700] "GET /showthread.php?t=1094814 HTTP/1.1" 200 14912 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.110 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=821678 HTTP/1.1" 200 18159 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=562761 HTTP/1.1" 200 10070 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.107.181 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=693491 HTTP/1.0" 200 53865 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:04 -0700] "GET /showthread.php?t=159640 HTTP/1.0" 200 55015 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:05 -0700] "GET /archive/index.php/t-537415.html HTTP/1.1" 200 3318 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:05 -0700] "GET /archive/index.php/t-563251.html HTTP/1.1" 200 1597 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.121 - - [31/Jul/2009:12:13:06 -0700] "GET /archive/index.php/t-1125505.html HTTP/1.1" 200 2069 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:06 -0700] "GET /itrader.php?u=13318 HTTP/1.0" 200 48510 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:05 -0700] "GET /showthread.php?t=786064 HTTP/1.0" 200 119848 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.119 - - [31/Jul/2009:12:13:06 -0700] "GET /archive/index.php/t-689564.html HTTP/1.1" 200 2521 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?p=12044091#post12044091 HTTP/1.0" 200 62587 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.111 - - [31/Jul/2009:12:13:06 -0700] "GET /showpost.php?p=7707267&postcount=34 HTTP/1.0" 200 6588 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:06 -0700] "GET /showthread.php?t=161620 HTTP/1.1" 200 14164 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.113 - - [31/Jul/2009:12:13:07 -0700] "GET /itrader.php?u=184293&iv=4 HTTP/1.1" 200 6730 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.112 - - [31/Jul/2009:12:13:08 -0700] "GET /showthread.php?t=134264&page=2 HTTP/1.1" 200 17335 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"
65.55.106.116 - - [31/Jul/2009:12:13:09 -0700] "GET /showthread.php?t=761369&page=3 HTTP/1.0" 200 70072 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)"

[DoDo Me]

Bing has much less data compare Google, so they must work harder, what do you expect them to do?

Get a better server or just block all robots except googlebot

Who has a better server may not win, but who has the worse server will definitely lose.

[digitalpoint]

It's not about needing a better server, the servers handle the traffic just fine. Just saying 8Mbit/sec wasted for a bad search engine that gives few users compared to Google isn't worth it, that's all. That's why I limited them via robots.txt. MSNBot was crawling at a rate of 1,287,360 pages per day, now they are allowed to get 43,200 per day. If they can start driving traffic at a crawl pages -> users sent ratio that's anywhere remotely close to Google, I would let them crawl more.

Right now MSN/Bing bring 0.2% of the search traffic Google does and waste about 40,000% more server resources than Google (those are real numbers, not made up BTW).

I think maybe the software engineers that designed Windows might have designed their search engine spider... Both are really bad pieces of software.

[Slincon]

i've heard about this before. msnbot also hammers requests to sites using various queries. There were a lot of complaints about this a year or two ago.

[addaminsane]

is it a good idea to block the msn bots as bing appears to be the up and coming online search engine?

[darkdrgn2k]

Always bad to block bots....

especialy if you wnt them to index your site

[webbynoc.com]

if they are a harm to the network i would block them, we do not really care about indexing we just keep our network safe and I would do such to prevent bandwidth wasting.

[digitalpoint]

Originally Posted by addaminsane

is it a good idea to block the msn bots as bing appears to be the up and coming online search engine?

Depends if you need to traffic or not. Search engine traffic ultimately comes at a resource cost (the resources used by the spiders). So if MSN can someday figure out how to drive more users for the amount of resources used, I'd let them spider more than the 40k pages/day I limited them to. Until then, they get throttled/limited here.

[anthonywebs]

wow i have never heard about this before... i hate bing anyway so... WOW 40.000% ??? thats amazing how hard microsoft tries to beat google but they never will

[Articlestopost.com]

Thins is the first time i am hearing about this. I will have to keep a close eye on my stats.

[menj1980]

Wow, that is really a lot of hammering involved. I better keep a lookout on my own forum in case this happens too.

[rustybrick]

http://www.bing.com/community/blogs/...s-at-bing.aspx

If you have any feedback for us on either the existing or the new MSNBot, or on any other matters pertaining to crawling your website, please post your comments and questions in our crawling/indexing forum. Our Webmaster Center forums moderator will follow-up with you ASAP. Please include in your forum post the identity of the user agent that is the source of your concern so we can help you that much faster. Thank you!

Hope that helps.

[FifthDimension]

Bing seems to be growing very fast, so I guess it is normal for them to send lots of read requests to get their index size larger than the competition.

[searchcandy]

RustyBrick for president!

http://www.seroundtable.com/archives/020524.html

[searchcandy]

Bing for the bin!

[Professional Dude]

I have seen Yahoo bot doing the same thing, but never seen Msn bot.

[digitalpoint]

It's not even that we can't handle the load. It's more like, "wtf is the point of letting them eat so much resources for so little in return?"

It's been 72 hours now since I added

User-agent: msnbot
Crawl-delay: 2

to our robots.txt file, and they have yet to start adhering to it.

Honestly, I think their search engine spider is just very badly designed. It seems like MSNBot causes problems for websites since it's inception.

Even in 2009, now they can't figure out how to throttle themselves, use HTTP/1.1 consistently or adhere to their own Crawl-delay directives. Thumbs up for Microsoft engineers! {rolls eyes}

[digitalpoint]

Oh, you can check out how well they are adhering to their own crawl-delay directive...

http://forums.digitalpoint.com/onlin...&pp=200&page=1

[Professional Dude]

Originally Posted by digitalpoint

Oh, you can check out how well they are adhering to their own crawl-delay directive...

http://forums.digitalpoint.com/onlin...&pp=200&page=1

136 bots only , i guess not so well.

[scylla]

820 bots when I checked, most were msn.