GoogleBot Slams My Site

[TwisterMc]

Googlebot came though my site eating up 3.12 Gigs worth of badwith. OMG I better do some directory blocking before they come back and eat up another three gig because then my site will be shut down cuz I only get 5 Gigs :eek:

[stephfoster]

Ouch... time for some quick decisions.

Badwith... good typo for when that much bandwidth is eaten so fast. Almost appropriate.

[Jackobo007]

Googlebot came though my site eating up 3.12 Gigs worth of badwith. OMG I better do some directory blocking before they come back and eat up another three gig because then my site will be shut down cuz I only get 5 Gigs :eek:

Go on securemate.com they give unlimited bandwith for very reasonable price.

[disgust]

no hosting providor can really offer "unlimited bandwidth." most "unlimited" bandwidth providors will force you to leave once you're using too much- usually around 100-200 gigs.

quality unmetered servers are the closest you'll get to unlimited, but even they are limited by the speed of the line they're connected to

[mxlabs]

3 GB of transfer in 9 days caused by googlebot alone seems to be impossible. make sure you have a look at the incoming IPs carrying the "googlebot" user agent.

[disgust]

assuming all of your pages were 100KB, it'd need to load about 30,000 pages to use that much. it's possible, depending on how many pages you have... but if you have THAT many pages, you're probably on a more serious hosting plan

if the average page was 50KB, it'd about 60K pageviews.

if it was 25KB, then about 120K pageviews

[TwisterMc]

I think it was due to a dynamic calendar which could technically go on forever. Making hundreds of dynamic pages ;)

[arestia]

i wonder if google has any sort of procedures in the bot to figure out that the site is generating infinate pages so it doesnt get into a loop forever.

-dan

[TwisterMc]

I wanna know how to control that. So, say, Google can only use up 2 gigs of bandwidth at a time. That'd be nice.

[bobafind]

Or you could just robots.txt that particular script (unless you want the actual pages with calendar content to get indexed).

Hmmm..

If you wrote the calendar script and know the inner workings, you could make a calendar archive page (like forums do) with "static" html links using mod rewrite and make it all SEO'd with keyword rich event titles and such. Then block google from looking at the actual, dynamic calendar.

Hey that's not a bad idea...

[TwisterMc]

I use PHPiCalendar that way it syncs my home calendar with the internet. :D I'm not writing my own script. Ohh no. ;)

Which reminds me, I should upgrade my calendar script.

[kuhleen]

how can you know how much each crawler consumes in terms of bandwidth?

[TwisterMc]

I have awstats (http://awstats.sourceforge.net/) installed

[TwisterMc]

Now a spider called Pompos slammed my site using 2.5 gigs worth of bandwidth. Good thing I bought more because I'm using more bandwidth this month than ever before. However, I don't understand how it could eat up that much since blocked my dynamic calendars. I just hope I can make it through the end of the month as bandwidth is expensive from this host. :rolleyes: