Huge bandwith used by google

Hello,

One of my sites, an real estate websites has in the last few days a very high trafic from google bot ( 6 Gb ) . A part of the website was optimized and the link structure was changed but for me it's a little bit strange this huge amount of trafic. I had ~48000 hits from google .

I think google still has the old files, can I be banned because of the high number of page in a short time .. or because some of the files are partialy dublicated ? The old files have noindex,follow atribute in metatags .

Now 2 weeks if using site:www-mysite on google I had ~14000 results and now it's going down .. I saw yesterday that the number was ~10000 and today ~9000 .

 

I highly doubt google used up 6gb of data from your site.

Also use the api and not the site command. I show over 1.5 million results for one of my sites using site: but only 86,000 using the api

 

I uploaded the awstats details for the robots hits, please belive me I did not had any reasons to lie .. this just happent and I don't knmow what to think about it.

Another proof that google visited this site much more than it usual does is that one the script which I made to track every visit of google it inserted ~19000 rows in my db

Format dynamic
Rows 19,240
Row length ø 114
Row size ø 115 Bytes
Creation Oct 29, 2005 at 01:28 PM
Last update Oct 31, 2005 at 03:11 AM

you can see all the rows ware added from 29 to 31 octomber.

Can you give me more details on how to use the api tool ?

 

Rows 19,891
Row length ø 114
Row size ø 115 Bytes
Creation Oct 29, 2005 at 01:28 PM
Last update Oct 31, 2005 at 04:03 AM

 

Do you got URLs that use session IDs??!!

In Googles own words...Session IDs suck.

 

on the old part of the site, yes I do have PHPSESSID .. but now I updated some pages and if useragent = google I will not ad sessionstart() .. I want to see if this is an solution but still there are alot of pages visited by googlebot which don't have the phpsesid variable in url.

 

I just do not understand how googlebot chews 6gb. I think someone is masquerading as googlebot and recursively sucking your site.

I get 8000000000000 times the hits from google bots and like not even close the bandwidth usage.

 

I know .. it's very weird for me too, yesterday I modified the script which insert in db every visit of googlebot and stored the user agent too, it's seems to be Google's useragent :

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

..later I will download the rawlog and take a look at the IP address

till then I uploaded the stats to see the changes ..9 Gb by google

!! I didn't see the file didn't upload because of the size .. no the images should be available.