Since the attacks of the Santy worms over Christmas, when my forum suffered a DoS outage as a result of the barrage of attempts to exploit a vulnerability that no longer existed, I have noticed that Googlebot activity on my forum has fallen way off: These stats are for the last three days: Bot / Number of Hits Media Partrners GoogleBot 760 MSN Search 562 Yahoo! Slurp 262 GoogleBot 16 Now Googlebot, since the forum launched last March, has always been #1 or 2 in this list (typically 1000+ hits per week) but the two things that have happened are that MSN and Slurp are both way up and Googlebot is way down since the Santy attacks. Note that Media Partners Googlebot (the AdSense spider) is still way up there, perhaps even increasing it's activity. Now, as a result of the Santy and other threats, I did add some code to plug the php vulnerabilities. It is also true that for periods of up to a few hours during the Santy bombardment my site was unusually slow to respond or even inaccessible due to the large number of hits from sites infected by the worm. Nothing else has changed. Note that I have separate stats for the forum and for my main site -- Googlebot is still very active on the main site (by the way, I do not have AdSense on the main site so no MediaPartners Googlebot there) so it's just the forum that has dropped off. Is anyone else seeing anything like this?
I never got hit by the worm but I've noticed a similar scenario myself. It seems to be all Yahoo and MSN at present although keeping in mind I've just installed the stats application again so this is only applicable for the last 2 days. The following is for my forum only. Yahoo = 361 MSN = 328 Google = 54 Even prior to this Google was well down to what the others were doing. As to why this is happening sorry I really can't explain it.
same stats on my sites. msn and slurp are pretty busy, googlebot visits much less than it used to be.
I wonder, assuming this is common (which of course has not yet been established): 1. does this indicate that Googlebot is paying less attention to spidering forum (and maybe blog) sites? 2. does it reflect generally reduced activity by Googlebot for some reason? 3. given that my forum includes AdSense and the main site does not, is there a connection there? Is Google putting MediaPartners Googlebot to double duty? (I realize this was not true in the past so interpret my question as "has this changed?") 4. is there something else going on here that is "none of the above"?
That was back in January. That pattern did continue for a while but Google is now back in force on my forums.