How can Google crawl a private forum? I know of a forum that has been crawled for over a year now and 99% of their forum is private, you have to register in order to view their content. But you can still see some cached content form google....They get around 20-30 new signups a day! How can I do this?
Google can crawl most forums (even if they are private) because the forum registers the Google Bot and then allows it to view all content. I believe that the only way to disable it is to delete the Google Bot as a user.
If you know some PHP, you can allow googlebot to crawl protected area. It is all about checking the user-agent and treating googlebot like a logged in user.
Well I noticed phpBB sets all bots to a certain usergroup allowing them FULL access to the forum even private sections. Does Vbulletin have the same?? If so, where can I get it??
Hmmm... Had to think through what you are asking... Why index private stuff...doesn't that make it NOT private anymore? Best Eric
you can actually set it in your admincp if you want that particular forum being indexed or not... well at least you can in vbulletin..
Would like google to see it but not unregistered guests, so this will make them sign up in order to view. Does that make sense? --John LaO, is I choose for it to be indexed, will it still be indexed if I make it unviewable to unregistered guests?
YEah thats why i have installed a plugin for firefox, Modify headers and i have modified my header to GOogle Bot's one. I can visit most of the forums without registering lol "P