I'm still trying to decide whether it's a good thing or not to block Google Bots from my vBulletin forum, or parts of the forum. Right now I have it completely open with no robots.txt file at all. What is your opinion and why?
I believe, for whatever site, if its necessary, block robots for potion of website is a good practice. save bandwidth avoid duplicated content
Okay, so then if you are going to block out sections of a forum, what sections of a vBulletin forum should be blocked in the robots.txt?
Oh, alright. I'm wondering what people do who run vBulletin because I have recently lost all my PageRank and I am doing anything I can to improve my crawls now!
That really depends on your own personal feelings and thoughts. On many forums, search engines are free to go just everywhere because webmasters of those boards like seeing more and more pages getting indexed.
It totally depends if your server can handle the load, but sometimes it is a good idea to block few parts of the site.
Would you block this part of the forum which is nothing more than RSS feeds from other websites being posted to the forum... I think maybe that is hurting my SEO. Not sure though. What do you think?
On my forum I have few sections that cannot be crawled by search engines because they are the places ONLY for members private discussions. Other than that all sections are open.