Is this right?

I was just noticing that google and yahoo are constantly spidering my site. Is this nescasary? Should I let them do so or is there a way that I can tone it down a bit?

Roger

 

Yes, it is good and necessary.
Just protect your secure areas and stuff you don’t want to be indexed.
You may do it with robot.txt file or meta.

 

Check the outcome, I don't think it is going to harm you anyway, just it might eat your badwidth.

 

Unless you let them spider your site, they won't reward you with traffic

 

For google spider, you can try slow down crawl rate at Google Webmaster Tools (https://www.google.com/webmasters/tools/) if you feel the spider crawl your site too often. It also provide other useful tools.

To adjust crawl rate, just go to Diagnostic -> Crawl rate.

 

This is essential especially if you have fresh content regularly

 

Yes I do have fresh content. I have lots of bandwidth so I'm not worried about that. I just noticed that they were there all the time. So I guess it's ok.

Thanks.

Roger

 

It certainly causes you no harm. You should be happy the search engines are taking so much interest in your site.

 

getting spidered regulary is very good

 

google bots is enough