My host is telling me the google bot is spidering my sites so much it is bringing down the server, they have blocked the googlebot at their firewall from coming into my sites. I am scrabmling here to try to satisfy them so they remove this block until I can find an alternative. How long can my sites go like this before google removes them from the index! (shudder) I have tried robots.txt, they said googlebot was ignoring it and spidering anyway...is the googlebot gone mad. Talk about starting off to a bad day.
Right...I wasn't happy at first they disabled all my sites, but upon my persistence they agreed to put them online as long as they could block google. I agreed of course, just to get them online again. I can't entirely blame them, as they are on a shared server, and the other folks sites were suffering because of my sites...but still. I feel so helpless...
There is no need to panic here. Your problem is simple..... Your hosts are idiots and have no idea how to run their business. I say this because of their lack of knowledge about the importance of that Google spider and that it is only temporary. Simple fix is to leave them immediately. You can: - find a new host - set up your account - change DNS - and move your site In less than an hour. Run and run fast!
One good thing...I am glad I have diversified my hosts. My sites are split across to completely seperate hosting companies. I guess dedicated is the only way to go if you don't want problems like this. :|
No half decent host would pull this...Something tells me you have "unlimited bandwidth" for your account Completely drop them as this is probably the tip of the iceberg. And there is nothing wrong with posting who it is as long as you are being completely honest.
Obviously, you haven't hit any limits, since the site was/is up - there should be no reason to block any visitor that is not malicious. All my big, well optimized customers are getting spidered like mad by Google of late. Most smaller sites are seeing some increase as well. Mind sharing your bandwidth use for the last week or two? Is Google really the big problem, or just the most obvious?
Month to date stats for 1 site. Googlebot 12822 hits 148.66 MB AskJeeves 1973 hits 22.48 MB MSNBot 581 hits 3.96 MB Code (markup): (from awstats) From this I can tell Google is way higher than any other bots....this is for a site that averages less than 200 unique visitors a day.
don't see why that would cause a problem as some of my sub domains are well above that .... do as others said and change host
148 meg and they shut you off, dude, find another host, there are FREE hosts that won't do this crap to you.
That was just a sample site, I have many others where google bot is doing major spidering. I know a host has to protect the other users on the server from latency, etc... but there has to be a better solution...