The logs on one of my new web sites show several deep crawls by a new? googlebot. Address: 66.249.65.179 Browser: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Protocol: HTTP/1.1 This bot eats up all of the pages but appears to have absolutely no effect in regards to indexing the pages in google search results. The "regular" googlebot Address: 66.249.64.16 Browser: Googlebot/2.1 (+http://www.google.com/bot.html) Protocol: HTTP/1.0 has so far only crawled the robots txt file and the index page of the web site which does show up in the serps. Does anyone know what the new bot does? Why the deep and persistent crawls if the indexing of multitudes of pages are not put into the google directory? Also the new bot - 66.249.65.179 - Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) does not show up in a web trends analysis of the vistor activity. I have to view the raw log to see it. Visits from the regular googlebot does show up in web trends in the browser area.
Haven't seen this bot yet on any sites. I can definitely say that my bot activity has been very demanding and fun fulfilled. Just about all of the bots (Google,MSN,Yahoo) visit every few hours lately Lots of Shaking and Dancin' goin on! KG
The best thing about it is it supports HTTP/1.1, and uses gzip compression for getting files... Some info over here.
it doesn't show up in my stats yet. but I have to say that it's a very robot-friendly week indeed. 300mb+ burned by them so far this month
2005-02-14 02:53:29 66.98.148.38 GET /index.htm - 80 - 66.249.64.4 HTTP/1.0 Googlebot/2.1+(+http://www.google.com/bot.html) - 200 0 0 20864 222.... I just see the regular bot in the logs, what's the difference to the new one? You are lucky for just burning 300 MB for bots, I used up 2 Gigs this month already
after having a closer look at my logs, I noticed it has been there for a couple of months... it really looks like somebody is spoofing his user agent, that's mainly why I overlooked it.