Spiders on my forum vB3.5.4

I have spiders on my forum, but I have the forum hiden to guests (I have some valid reasons for this at the moment, it isnt something I particuarly like doing but needs must) Anyway, the spiders seem to be treated as guests, is their a way to change that so Spiders are seen as members, that way they get through the forum. or will it not matter much?

 

You could blacklist the user_agent; if it does not contain MSIE/Safari/Firefox/Netscape you could log that session in as a member.

But note that if this results in a bunch of forum pages being indexed, what happens to a surfer who clicks on a link in a search engine? He is requesting a page that he doesn't have access to.

 

Good point, but I have my reasons that outweigh that issue, I dont intend to have the forum hidden for long, a few weeks at the most, but the spiders have been pretty active recently and I dont want to have a load of search results that are the no access message.

 

This is a good point. You would need to do something like some of the major newspapers who pull this s**t. Hopefully they can be automatically directed to hit "back" and then read the cached version i.e. instead of being forced to sign up.

Eric

 

In vBulletin, if you have things set properly, a guest/spider cannot see those forums.

Check your forum permissions.

 

I know, but I want the spiders to see the forum and not be treated as a guest.

 

Then change the forum permissions so guests can view it.

You cannont have spiders see the forums, and not regular people.

 

No, as I said above, I have reasons for not allowing the guests to see the forum so I will not change that.

I wasnt sure if I could allow spiders to be treated as members or not, thanks for clearing that up.

 

Not without a hack - and that would be considered cloaking by google, and you would likely get banned from the index completely.

 

Yes, this can be done...with the following -

http://www.vbulletin.org/forum/showthread.php?t=108070

 

Don't employ cloaking or sneaky redirects.

Make pages for users, not for search engines. Don't deceive your users or present different content to search engines than you display to users, which is commonly referred to as "cloaking."

Ref: http://www.google.com/webmasters/guidelines.html

 

hehe, that's one way to get a competitor out of the SERPs, teach them cloaking. Very clever!

 

Who got red repped in this thread? Someone gave red rep to me saying "negative rep for answerin the creator's question?" -- ??

 

Hmmph, I didn't realize that I was in competition with Motorsports