Search Engine Bots Bandwidth Usage

Has anyone else seen anything like this?

Inktomi Slurp 805+247 7.18 MB
MSNBot 434+55 13.66 MB -what the?! 13MB? and half the crawl of IS
Googlebot 137+13 3.96 MB -I love google, what a small footprint!

Anyone else notice how much bandwidth MSN uses when crawling?

 

I've had bots take up as much as 3GB The issue was with a dynamic script. Technically it never ended so the bot just kept going and going and going ...

 

yea I had the same... when bots are poorly coded they just crawl your site to death. I had a "sysinfo" script on my server which dynamically linked to pages stating the server usage and stuff and would produce new links every run. Basically I ended up with thousands of indexed pages in msn and other engines, not google though

 

Yes I have!!

Check this out, January figures :

Inktomi Slurp 1709+740 50.26 MB
LinkChecker 992+16 40.19 MB
MSNBot 836+62 13.15 MB

Anyone else having Inktomi going mad?
And what the hell is LinkChecker?

 

This is from webalizer for a new site I just put up last month... check out the spider activity on these (see image)!

I went over bandwidth just from spiders!

It is crazy!

 

Heh, here's this months info for my site for the big engines:

MSNBot 14682+125 534.38 MB
Inktomi Slurp 1879+350 33.24 MB
Googlebot 7612+135 270.10 MB 10 Feb 2005 - 19:13
AskJeeves 95061+72 11.23 GB
Alexa (IA Archiver) 345+89 10.36 MB

Check our the AskJeeves usage. (11.23GB in 11 days!)

This is actually less than normal, as GoogleBot generally caches 100k pages a month from me, and MSN is usually close behind. (I don't mind, since they both use HTTP1.1 and don't use much bandwidth.) AskJeeves seems to only be grabbing my page HTML and then my Flash file. I think I'm going to restrict its ability to grab the Flash, so I can lower the bandwidth usage.

DS

 

Googlebot 1711+24 17.70 MB 11 Feb 2005 - 21:52
MSNBot 1632+95 16.98 MB 11 Feb 2005 - 21:55
Inktomi Slurp 347+144 3.34 MB 11 Feb 2005 - 20:34
Unknown robot (identified by 'crawl') 141+3 640.64 KB 11 Feb 2005 - 17:48
Unknown robot (identified by hit on 'robots.txt') 0+59 13.42 KB 11 Feb 2005 - 21:32
AskJeeves 14+14 145.82 KB 11 Feb 2005 - 21:16
Alexa (IA Archiver) 8+10 56.58 KB 11 Feb 2005 - 11:45
Unknown robot (identified by 'spider') 8+6 82.21 KB 09 Feb 2005 - 02:18
WISENutbot 3 30.03 KB 01 Feb 2005 - 10:57
Unknown robot (identified by 'robot') 0+1 237 Bytes 06 Feb 2005 - 16:39

For my semi-unpopular site.

Wow, AskJeeves attacked you, darq. Googlebot hit 1000 times during the update over the passed few days, Googles usually 1/2 or less of MSNs hits, but usually nearly equal bandwidth.,

On a side note, Inktomi hits Robots.txt waaaaaaaaaay too much for the 0.00 times I've updated it since I launced the site.

 

This has been concerning me actually. Not the bandwidth usage - but WHERE IS GOOGLE?

Here is December

MSNBot...........13911+183.......390.08 MB
Googlebot........11850+82.........276.62 MB
Inktomi Slurp......8310+1586.......83.11 MB


Now here is January

MSNBot..........21545+315......607.31 MB
Inktomi Slurp...15723+2329.....157.52 MB
Googlebot.......13779+48.........216.13 MB


And so far, Febuary

MSNBot.........................................8419+85......227.39 MB
Inktomi Slurp..................................6773+761......65.22 MB
Unknown robot (identified by 'spider')..1609+36........50.81 MB
Googlebot.......................................1504+21.......44.65 MB


MSN has gone totally nuts, so has yahoo. Google seems to be holding back.
The other strange thing - Look at January - Inktomi Hit more then google, but used less bandwidth???

-

 

Dynamic content can get a bot stuck if it's never ending. Like a calendar that can go for years. So could a bot

 

3GB! God I'd throw a temporary robots file at that in a heartbeat.

I'm too poor for such wastes in BW.

 

It's been fixed.

 

could somebody point me in the right direction for finding which ip addresses belong to search engine bots?

thanks

 

This months stats ...

PurePimps:

Googlebot 63911 471.14 MB 26 Feb 2005 - 08:24
MSNBot 24554 686.09 MB 26 Feb 2005 - 14:46
Inktomi Slurp 11449 95.01 MB 26 Feb 2005 - 14:21
Alexa (IA Archiver) 762 4.87 MB 26 Feb 2005 - 09:26
Unknown robot (identified by 'crawl') 479 4.26 MB 26 Feb 2005 - 11:50
Unknown robot (identified by 'spider') 297 2.55 MB 26 Feb 2005 - 05:45
Unknown robot (identified by 'robot') 179 2.23 MB 26 Feb 2005 - 02:22
AskJeeves 38 642.65 KB 26 Feb 2005 - 10:56
LinkWalker 17 227.45 KB 25 Feb 2005 - 13:53

W3Dictionary:

AskJeeves 196698 375.46 MB 26 Feb 2005 - 12:54
Googlebot 55963 182.64 MB 26 Feb 2005 - 09:16
Inktomi Slurp 40246 134.30 MB 26 Feb 2005 - 14:34
LinkWalker 9392 182.95 MB 25 Feb 2005 - 07:21
Unknown robot (identified by 'crawl') 2562 7.01 MB 23 Feb 2005 - 19:22
MSNBot 133 15.30 MB 26 Feb 2005 - 05:11
Unknown robot (identified by 'spider') 95 26.12 MB 26 Feb 2005 - 06:50
Alexa (IA Archiver) 79 11.26 MB 26 Feb 2005 - 12:47
Walhello appie 26 1.62 MB 18 Feb 2005 - 18:58
Unknown robot (identified by 'robot') 3 7.53 KB 22 Feb 2005 - 18:47

AZ-SONG-LYRICS:

Googlebot 68354 272.90 MB 26 Feb 2005 - 03:45
LinkWalker 6906 31.41 MB 18 Feb 2005 - 08:30
Inktomi Slurp 1659 8.07 MB 26 Feb 2005 - 02:58
MSNBot 1599 7.79 MB 26 Feb 2005 - 06:22
Alexa (IA Archiver) 326 4.50 MB 25 Feb 2005 - 18:11
Unknown robot (identified by 'crawl') 194 1.96 MB 22 Feb 2005 - 05:32
Unknown robot (identified by 'spider') 51 744.42 KB 26 Feb 2005 - 02:32
Unknown robot (identified by 'robot') 29 403.76 KB 19 Feb 2005 - 12:14
Walhello appie 27 381.54 KB 19 Feb 2005 - 04:08
AskJeeves 25 257.26 KB 26 Feb 2005 - 10:27

 

Have you done any comparisions with the total "bot" bw usage vs. your total bw usage used to simply deliver content from the pages? It would be interesting to take all of that into account to see if they bw costs/usage eats into any profit made from the sites themselves. I know we have looked at what it costs to develop and maintain a site and what our costs are in terms of bw, storage space, etc., and have found where the profit margins are. I just have never really taken the bot traffic into account. Anyway, that would be an interesting thing to look at.

 

I took a look at last months bandwidth usage, and bot bandwidth usage and it's less that 5% of total usage. So I can live with that, if that's the same or better for other users I don't see anyone having any problems with the bandwidth bots are using. Now if these bots start using more bandwidth than our users are taking up then we've got a problem and I think search engines will have to rethink thier strategies.

 

AskJeeves 111343+50 724.56 MB 13 Mrt 2005 - 17:59
Inktomi Slurp 17469+996 125.18 MB 13 Mrt 2005 - 17:59
MSNBot 13531+64 396.37 MB 13 Mrt 2005 - 16:31
Googlebot 10840+71 208.49 MB 13 Mrt 2005 - 11:40
Google AdSense 8777+19 63.56 MB 13 Mrt 2005 - 17:54
Alexa (IA Archiver) 2217+29 61.73 MB 13 Mrt 2005 - 17:59
Webinator 2123 94.96 MB 13 Mrt 2005 - 15:27
GigaBot 1891+60 51.47 MB 08 Mrt 2005 - 05:40
Unknown robot (identified by 'crawl') 344+1 8.59 MB 09 Mrt 2005 - 05:45
BaiDuSpider 129+2 8.20 MB 13 Mrt 2005 - 13:40

 

the most interesting think is .. that this unknown robot did the huge traff on my new site

Unknown robot (identified by 'crawl') 112525+6 3.63 GB 19 Mar 2005 - 00:42
Googlebot 26200+113 858.58 MB 20 Mar 2005 - 06:06
MSNBot 8383+120 284.41 MB

 

# kb site
1 1317387 crawl-66-249-65-70.googlebot.com
2 1144040 crawl-66-249-65-207.googlebot.com
3 620728 ip24-250-11-107.ri.ri.cox.net
4 479172 msnbot.msn.com
5 309907 crawl25-public.alexa.com
6 227129 crawl-66-249-65-101.googlebot.com
7 216793 crawl-66-249-65-228.googlebot.com
9 102410 crawl-66-249-66-144.googlebot.com
over 3.5gb last month :x

 

heres mine

Inktomi Slurp 3323+490 71.42 MB 31 Mar 2005 - 09:44
MSNBot 1847+234 47.23 MB 31 Mar 2005 - 09:35
Googlebot 414+48 8.45 MB 31 Mar 2005 - 09:51

yahoo seems to have hit my site hard this month

 

I saw

Google AdSense 808+31 7.63 MB 31 Mar 2005 - 08:03
Inktomi Slurp 88+37 418.13 KB 31 Mar 2005 - 07:08
Googlebot 89+9 615.87 KB 29 Mar 2005 - 21:11
MSNBot 33+3 354.29 KB 16 Mar 2005 - 07:39
Alexa (IA Archiver) 25+10 178.03 KB 14 Mar 2005 - 11:54
Unknown robot (identified by hit on 'robots.txt') 0+31 1.82 KB 31 Mar 2005 - 00:00
LinkWalker 27+1 232.07 KB 08 Mar 2005 - 02:43
BaiDuSpider 15+2 103.05 KB 26 Mar 2005 - 05:20
AskJeeves 9+5 65.98 KB 25 Mar 2005 - 00:57
Unknown robot (identified by 'robot') 4+4 28.74 KB 29 Mar 2005 - 04:53
SurveyBot 4+4 39.97 KB 28 Mar 2005 - 00:46
MSIECrawler 6+1 109.25 KB 04 Mar 2005 - 17:12
Pompos 1+2 9.73 KB 03 Mar 2005 - 13:42
Unknown robot (identified by 'spider') 1+1 10.98 KB 31 Mar 2005 - 17:31
ObjectsSearch 1+1 9.67 KB 21 Mar 2005 - 08:19
Unknown robot (identified by 'crawl') 1 10.76 KB 30 Mar 2005 - 09:41
Netcraft 1 0 30 Mar 2005 - 17:29

last month. bots and spiders acounted for 10% of all my bandwidth and close to 20% of my hits, almost all of that being adsense.