I got this when you login to - https://www.google.com/webmasters You have to verify your website before you can change the crawl rate Here is a screenshot of google saying that the rate should be Normal. Thanks Ishan
They are just telling you what to use as the default in the absence of other factors. There are other factors such as the bot getting confused as I pointed out above where slowing the bot is appropriate. I agree that a normal crawl rate should be used under normal circumstances. There is nothing wrong with slowing the bot when there is a problem as there was with the original poster of this thread. That is why there is the other option.
Dreamhost has always been a pain in the ass. My only site with them (one I bought) is down constantly. It earns some money for me - my best site - and it dropped in PR from 5 to 4. I'm wondering now if this may have been a factor. I wouldn't have received the email - it would have gone to the owner before me, or before him. Well, I've sold the site and I will never use Dreamhost ever again. Not by choice, anyway.
OHH MY GOD.... F**K dreamhost I just found out same thing on my domain and shit its screwing my SERPS, no wonder.
This is a nightmare. I start looking for a good hosting company and found this thread. Now I will check many hostings at forums first, to see what people say. Last time I was screwed up by some reviews. Don't trust reviews
I'm glad Michael stopped by and squared things away. I have 5 blogs and 2 sites with them and have for the last 3 years. I also have 8 accounts with other hosting companies and Dreamhost has consitently been better with customer service and hosting my sites than the others. Bluehost would be my 2nd pic as a provider that can come close to matching Dreamhost IMHO. The others talk a good game but fail to produce. TxDon
if dreamhost has said this, dont believe it they are the worst hosting company. if you add that line to your robots.txt file, all of your pages will be blocked
Old topic but unfortunately for me, Dreamhost apparently keeps going with it's unfair method After almost a 40% traffic slump in 10 days (and counting), i realised Dreamhost blocked googlebot via my site .htaccess. $ cat .htaccess <Limit GET HEAD POST> order deny,allow deny from 193.47.80 deny from 66.249 </LIMIT> I was ranking v/ well for popular keywords and most of my pages (include frontdoor) have now all disappeared from google (i guess the whole site will soon be deindexed as google updates its links and hits 403 Forbidden). The only feedback i got so far from Dreamhost is "I am assuming you prefer that I block the bot to disabling your account or domain but if you prefer I can do that instead." Anyone had this problems recently? I guess i have no other alternative than moving away from DH.
This behaviour really is unacceptable. What they should do is ask you to add the specified .htaccess or robots.txt file yourself. As a last resort, they could add it themselves and email you to let you know that they have done it. Modifying your site and/or blocking parts of your traffic, no matter where that traffic is from without notifying you is completely unacceptable. Up until now I had only ever heard good things about DreamHost. I like that they are participating in the discussion but that behaviour needs to change. Remember, if you aren't satisfied, you can always vote with your feet.
Hi I am a Dreamhost users almost 2 year. From the cost-performance point of view, I think they are good enough. I also got an email from them today complain my sites having the same problem (search engine BOT hits too high). They are asking me to do something. I am now looking into their info and trying to make my sites working fine. That's also the resaon why I visitng this sub-forum trying to collect more info. And I found this thread which is also informative. I really want to work with my webhost to get things "happy"......