I now have a bit of a problem with Googlebot and would like to hear other experiences to see what to do next: a) i got a dbase driven site, it's possible to 'see' 3 million different urls if you want to b) googlebot wants to crawl them all - or so it seems to c) it has now an open connection with my server(KeepAlive) and requests a page directly after another one - no delay and therefore causing a huge load on the server. d) all this together: Gbot uses 1mbit, 5 pages per second, for days in a raw now. I like to have a lot of pages indexed, I do get more referres from google, but I soon have to buy an extra server, just to server Googlebot requests... now the sollutions: someone mentioned emailing google and asking them to slow down the bot. Threads like this end with: "Googlebot doesn't show up anymore ...", that's not what I want. What I've done so far to get server loads back to normal settings is to have a " SLEEP(0.5 seconds) " after each Googlebot request (based on IP), that seems to work for now. Also I've seen robots.txt with "Crawl Delay: 5" which will keep Y!Slurp and Teoma spiders away for 5 seconds after each query, but Googlebot does not understand the Crawl Delay line. any ideas on how to slow GBot down , just a little bit?
I had a problem like this a while ago, my host e-mailed google and they said they would slow the bot down, they did and I'm still getting a lot of visits from then. It dropped to only a couple of hits the few days following the e-mail but it's gone back to around 70 a day now.