Googlebot is following me...

I’ve noticed an interesting behavior in googlebot… basically; it detects when I enter a website I own and visits the exact same pages I visit.

I can track this, just because I have a fairly new vB forum and currently very few people visit it, as it is less than a month old and still in the sandbox. Therefore, when I enter, I can see that there are no users online… just me. Ten seconds later, there are 2 users online. Then I go to the “Currently Active Users” page, and I see (as I’m the Administrator) that the other visitor is a googlebot. Then I move around the forum, and stay for a few seconds in a specific forum, an then go back quickly to see who’s online, and amazingly, every single time, the googlebot is viewing the same page that I was viewing.



After I refresh the “Currently Active Users” guess what… googlebot is back there too! Then I go to another page and do the same thing as before, and again, every single time I do this test, googlebot follows me around.

I don’t know if I should consider this a positive or a negative thing, but it is definitely interesting.

I’ve heard of bots trying to learn user behavior patterns, but had never seen such an evident prove of this.

I would like to hear your thoughts about this.

 

Hmmm, I wonder if this is related to the info they gather via the toolbar? Do you have the Google Toolbar installed?

 

No, I removed the toolbar months ago, because I had noticed that it was somehow tracking my movements and I didn't completely like it, although they at least acknowledge they do.

http://toolbar.google.com/privacy.html

 

I too have found i can put up a new page, check it for errors before loading the pages that link to it, and googlebot often goes to the page with no links to it yet.

I have always summed that it was finding the pages from the toolbar and that it did that for all toolbar users... but maybe not.

 

Are you sure it's "Googlebot/2.1 (+http://www.google.com/bot.html)"

and not

"Mediapartners-Google/2.1"?

The mediapartners is google adsense.
This has the same activity on my board as you describe.

In your vb setup in "who's online options" tab, change one to "mediaparters-Google/2.1" and create another that says "Googlebot".
Put the mediapartners fist.

See if you get the same results.

 

I did what you suggested, but don't see any change.

The complete ID of the bot that is currently chasing me is:

crawl-66-249-65-43.googlebot.com

 

Google-guys are great. I heard they created department for chasing people. It's about 1,000,000 new job places and supported by George Bush government program. They track actions of those people who uninstalled Google Toolbar, they check for their IP and with the help of FBI and Interpol brake into your homes and set up bugs and small i-cams. So no whander you see such an activity.

If seriosly, or you have Toolbar and then your suggestion can be true, or listen to Shawn, it's adsense bot.

 

I failed to mention... change the mediapartnrers to "GoogleAds" or something in the spider identification area so that you can distinguish the difference in the "who's online" page.

Also, as shawn requested... to find out what the Agent is, look at the line in your access logs.

True googlebot:
crawl-66-249-64-131.googlebot.com www.debatepolitics.com - [15/Nov/2004:01:53:56 -0500] "GET / HTTP/1.0" 200 33065 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"

or

GoogleAds bot:
crawl-66-249-65-201.googlebot.com www.debatepolitics.com - [15/Nov/2004:12:27:37 -0500] "GET /showthread.php?p=1586 HTTP/1.1" 200 10964 "-" "Mediapartners-Google/2.1"

The last part between the quotes is the Agent.

 

You were right Vauge. Good tip.

It is the Mediapartners-Google/2.1. However, it’s still interesting how they’re able to track your activity.

I don’t think they track everybody’s movements, but maybe if the site is publishing AdSense, they’re just checking user’s behavior in order to eventually determine if the clicks are legitimate or not.

I’ve heard that they try to determine if it is actually a visitor who is clicking the Ads or if it is some kind of automated program.

I’ll keep an eye on this and see what happens. Anyway they actually collect that kind of information, maybe not on a “person-by-person” basis, but more as a marketing tool.

“Google collects limited non-personally identifying information your browser makes available whenever you visit a website. This log information includes your Internet Protocol address, browser type, browser language, the date and time of your query and one or more cookies that may uniquely identify your browser. We use this information to operate, develop and improve our services.” Google Privacy Policy - Version 07/01/2004

 

That makes a lot of sense, since, as I said in my first post, my site hasn’t been left out of the sandbox yet.

Thanks for the Enlightenment!

 

I was just reading Uncle Johns Unstoppable Bathroom Reader - specifically the part about the conspiracy theory surrounding the U.S. moon landing. Then I popped online and spotted this thread right away. I love it!

 

Tapanti, can you visit some of my sites, Please.

 

LOL!

I might have found a new business here!

Googlebot Bait!

 

Its just so I see your "cough" problem 1st hand.

 

Hi Darksat,

I wouldn´t worry about Googlebot here. Seems to me that you should worry about ALL the other major bots.

Googlebot has already visited your site and in fact, there are already 17 pages indexed in Google´s index, but you have 0 "cero" pages indexed in all the rest of important SEs, such as Yahoo, Alta Vista, MSN, etc.

Those are the ones you should focus on now.

Googlebot will find its way to the rest of your pages.

However, I have to confess that seems weird that Googlebot has indexed your pages while all the others haven´t. Are you doing something to prevent this on purpose?

.

 

No, googles just being fast, site hasnt even been up 2 weeks.

 

That is even weirder. Your domain name was registered less than 3 weeks ago and has 17 pages already indexed in Google?

I guess the so called "Sandbox Effect" didn't apply to your site. Good for you. I wonder if it is because it is not a .com but a .co.uk domain.
.

 

Pages havnt been cached yet and ive added about 1000 backlinks.

also the "sandbox" wont stop your pages getting spidered.