Why does Googlebot visit twice?

I have been noticing for about the past week, that Googlebot visits most, but not all, of my sites twice each time.

The bot will visit and then about 8 minutes later, I will get another visit.

Previously I only used to get one visit.

Is this happening to anyone else?

 

Same here, eight minutes just like you mentioned.

I do not know what that is about. I'm hoping it's something good.

 

The first one is the supervisor bot: he doesn't actually do any work... he just peeks in and says, "Get a team in here, stat!" and then goes for coffee... then 8 minutes later, the team shows up.

How often does the content on your site change? What sort of site is it? Are you using a lot of scripting? Redirects?

 

Not quite sure what you are seeing Ned, but it is not uncommon for the bots to visit a site many times in a day, but this has been complicated with a new bot that Google have running around, and it may be that you are in fact seeing two different bots on two different missions.

I'd suggest a bit closer inspection of your stats and determine if they are the same bot each time, how many pages they took each time, if the queried the robots.txt file each time, etc.

 

It's also not unusual on a large site or forum to see several googlebots in at once, or within a few seconds of one another...

 

The sites are changind daily, but don't have any redirects. It's dynamic content like the Coop Ads that are causing the daily change.

I'll look and see if it's the same bot on both visits.

 

Googlebot checks if your site is not generated dynamicaly.

Tomecki

 

Tomecki, Googlebot spiders dynamic sites just fine as long as the URL string doesn't contain a long list of variables or the site requires session IDs (cookies) or login -- in any case, that's not the problem because the spiders are coming and apparently returning.

Badlay, as Mel indicated, Google now has two types of bot scooting about the net and there has been speculation here on this forum and on others about what new features or new purposes the newer version may have -- that's why I was curious about the type of site and pages, because some of the speculation is about whether the new bot has better javascript capabilities or whether it is searching out "cloaked" sites or redirected sites. So you might check your logs to see if this is the same googlebot version or both versions you're seeing.

But in any case, really it is not unusual on sites, especially those that change frequently (new content), for googlebots to come in groups. On one of my forums, we have seen on several occasions 12+ googlebots in at the same time. I don't think this is anything to worry about -- in fact, it's probably something to be happy about.

 

From the latest Search Engines Facts newsletter:

I'm not sure I agree with all of this speculation but time will tell...

 

Thanks for the info, Minstrel.

I've learnt quite a bit from this forum over the last few months, but not enough to understand how to do cloaking, so as I don't use any cloaking, it appears I don't have anything to worry about.

 

I would like to ask two follow-up questions, one concerns Session Id data and the other is cloaking.

How much is too many characters in a session id?

I’ve read a thread in this forum where someone once said that in some cases cloaking is ok. What exactly is cloaking? From reading these forums, I determined that clocking is presenting of one version of the website to a spider and another to a user.

I have a dynamic website. Accessing some areas of the website requires someone to push a button. Since the spider can’t push the button, we show a different version of the site to the spider. The casual visitor probably couldn’t tell the difference. Using two versions is our method of making sure the spiders see all our products. Is this an acceptable use of clocking? In my opinion, this is not an abuse of a questionable SEO tactic.

We’re not showing a version of the website stuffed with keywords.

What’s your opinion?

 

I think search engines catch cloaking by fetching the web page with a "disguised" browser tag and then comparing it to the data fetched using the "normal" browser tag. Now, since both spiders cannot push the button they will see the same version and you won't be penalized.

Search engines might use a manual approach but then it gets expensive and time consuming. If you the difference in the page content is legitimate (seems so in your case) then again the page will not be penalized.

Why don't you consider putting a text link or an image link instead of the button? This will make sure spiders and humans see the same version!

 

That was my question, too... why is there a need to click on a button at all?

 

Googleguy once stated that anything with more than ten characters in the session ID might be a problem.