They are SUCKING my bandwidth grrrrr.... 66.218.65.40 282 282 27.70 MB 14 Feb 2006 - 06:56 216.109.121.71 174 174 87.33 MB 14 Feb 2006 - 06:56 216.109.121.70 122 122 61.23 MB 14 Feb 2006 - 04:27 209.18.119.138 98 98 25.83 MB 14 Feb 2006 - 03:54 206.188.0.10 97 97 25.98 MB 13 Feb 2006 - 23:39 206.188.0.11 80 80 19.88 MB 14 Feb 2006 - 01:17 209.18.119.137 74 74 18.82 MB 13 Feb 2006 - 20:03 68.142.230.49 27 27 13.55 MB 13 Feb 2006 - 21:46
For the second month I have 66.249.72.206 which is a Google IP sucking down 8 gigs so far this month, on a site where the normal traffic is about 50 megs per day. I wrote them about it last month and got a reply telling me how to slow down their spider. I guess I wasn't clear enough that speed was not the issue, total bandwidth was. Most days show my usual 50 megs but there are four where as much as 2.5 gigs was used in one day. 8 gigs is enough to copy the entire site 75 times! Hate to block Google but not sure what else to do.
You can create a robots.txt file and place it in your root level User-Agent: Googlebot* Allow: / Disallow: /cgi-bin Disallow: /images Disallow: /any_directory_or_file_that_does_not_need_to_be_indexed more information at: http://www.google.com/webmasters/bot.html