Ok, I've been looking at my stats for the past 14 days, and it's extremely wierd on how google compared to MSN hit my sites. Googlebot will hit my site in intervals of two, and then leave...ALWAYS. MSNBot will hit it at least 15+ before leaving. AWSTATS: Googlebot 188 3.64 MB 15 Apr 2005 - 00:26 MSNBot 1928 42.07 MB 15 Apr 2005 - 00:29 Has anyone else had this happen to them?
Well i wouldnt be to much worried how google crawls, as long they index my pages its fine with me. However if you like to read a bit more about it then this article might be of any interest: Timing Google's crawl
After reading the article, perhaps MSN has the same concept, and their deep crawl is doing my site now, they have been hitting it non stop for the past three hours. Hopefully google will do a deep crawl soon.
google is way behind as far as hits go but still seem to index just as much. MSNBot 1297+179 24.78 MB 15 Apr 2005 - 10:41 Inktomi Slurp 944+262 14.97 MB 15 Apr 2005 - 08:56 Googlebot 699+42 17.40 MB 15 Apr 2005 - 10:41
Lol that doesnt say anything those statistics. According to my stats MSN is way behind Google AdSense 30923+32 212.08 MB 14 Apr 2005 - 18:59 Inktomi Slurp 19089+1272 142.88 MB 14 Apr 2005 - 18:59 Googlebot 19839+113 543.89 MB 14 Apr 2005 - 14:37 MSNBot 9169+98 262.54 MB 14 Apr 2005 - 18:59
Googlebot's behavior has changed pretty dramatically looking back the last several months. Googlebot basically crawled as much as it can and created index in the past. It seems that Googlebot crawls selectively today.
It seems like google is ALWAYS at my site. These numbers go up every day, but the G numbers go up a lot faster. Page count changes every day too. Lately it's been on a solid upward trend. It's interesting that Inktomi has been around almost constantly the past week as well, and has finally got more than 10 or 15 pages indexed. That never happend before. Also notice that while MSN is only around about half as much as G, it uses a lot more BW! Googlebot 2119+1198 11.09 MB 15 Apr 2005 - 08:16 MSNBot 1012+85 18.95 MB 15 Apr 2005 - 08:13 Inktomi Slurp 386+236 8.53 MB 15 Apr 2005 - 07:14
LOL, I get more bandwith used by bots then users. I'm not complaining just wish they would spider some of my other sites as well as the spider my main site. Googlebot 5462+58 118.46 MB 15 Apr 2005 - 17:17 MSNBot 4907+107 114.14 MB 15 Apr 2005 - 17:47 Inktomi Slurp 174+106 3.95 MB 15 Apr 2005 - 17:37 My new amazon http://buybooks.saltwaterflyfishers.com site is MSNBot 227+10 6.14 MB 15 Apr 2005 - 02:14 Alexa (IA Archiver) 22+5 1.12 MB 09 Apr 2005 - 22:55 Googlebot 10+8 187.27 KB 14 Apr 2005 - 22:31
Googlebot generated a few Gigabytes in a couple of days on my lyrics site. That's about 10 times the traffic I get from normal visits
Google hits my old sites 10x more often than msn... MSN hits my new sites 10x more often that google... different strategies for each SE
Google keeps on shoveling the directories up and down my older sites like 30-50 minutes every day, while newer sites might just get two pings a day. I don't find a regular pattern of MSN though, even less corresponding to some Google-manner. MSN hits most of my sites like fifteen times a day in fragments, picks pages in different directories, no real pattern to me