Hello, One of my sites, an real estate websites has in the last few days a very high trafic from google bot ( 6 Gb ) . A part of the website was optimized and the link structure was changed but for me it's a little bit strange this huge amount of trafic. I had ~48000 hits from google . I think google still has the old files, can I be banned because of the high number of page in a short time .. or because some of the files are partialy dublicated ? The old files have noindex,follow atribute in metatags . Now 2 weeks if using site:www-mysite on google I had ~14000 results and now it's going down .. I saw yesterday that the number was ~10000 and today ~9000 .
I highly doubt google used up 6gb of data from your site. Also use the api and not the site command. I show over 1.5 million results for one of my sites using site: but only 86,000 using the api
I uploaded the awstats details for the robots hits, please belive me I did not had any reasons to lie .. this just happent and I don't knmow what to think about it. Another proof that google visited this site much more than it usual does is that one the script which I made to track every visit of google it inserted ~19000 rows in my db Format dynamic Rows 19,240 Row length ø 114 Row size ø 115 Bytes Creation Oct 29, 2005 at 01:28 PM Last update Oct 31, 2005 at 03:11 AM you can see all the rows ware added from 29 to 31 octomber. Can you give me more details on how to use the api tool ?
Rows 19,891 Row length ø 114 Row size ø 115 Bytes Creation Oct 29, 2005 at 01:28 PM Last update Oct 31, 2005 at 04:03 AM
on the old part of the site, yes I do have PHPSESSID .. but now I updated some pages and if useragent = google I will not ad sessionstart() .. I want to see if this is an solution but still there are alot of pages visited by googlebot which don't have the phpsesid variable in url.
I just do not understand how googlebot chews 6gb. I think someone is masquerading as googlebot and recursively sucking your site. I get 8000000000000 times the hits from google bots and like not even close the bandwidth usage.
I know .. it's very weird for me too, yesterday I modified the script which insert in db every visit of googlebot and stored the user agent too, it's seems to be Google's useragent : Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) ..later I will download the rawlog and take a look at the IP address till then I uploaded the stats to see the changes ..9 Gb by google !! I didn't see the file didn't upload because of the size .. no the images should be available.