Robots.txt, Vbulletin & Google

[Design Agent]

Are there any recommended pages that vbulletin creates that should be blocked from google ?

For example - I have 'send pm' pages showing in my site: search. which are not necessary at all.

Any other I should block specifically?

[digitalpoint]

What version of vBulletin are you using? vBulletin 3.5 does a really good job at using nofollow on stuff that shouldn't be indexed right out of the box.

[Design Agent]

Version 3.5 (though I need to upgrade) to 3.51

Pages like this are being indexed (I haven't checked to see which others yet)

/forums/private.php?do=newpm&u=40
/forums/private.php?do=newpm&u=24
Which are coming up as supplemental and not necessary.

[digitalpoint]

If they are supplemental, it's probably really old (before you upgraded to 3.5). I wouldn't worry about it since new ones won't get indexed and the old ones shouldn't get reindexed. But if it bugs you, just add private.php to your robots.txt exclude directives and get them out quick with this:

http://services.google.com/urlconsole/controller

[Design Agent]

If they are supplemental, it's probably really old (before you upgraded to 3.5).

Actually when I upgraded I moved the forum to a different address and used google's page removal tool for the old address to deindex all the pages first (kind of a mini wmw for that section), so its not that I guess.


I wouldn't worry about it since new ones won't get indexed and the old ones shouldn't get reindexed. But if it bugs you, just add private.php to your robots.txt exclude directives and get them out quick with this:

http://services.google.com/urlconsole/controller

Thanks for the tips. I'll just leave it if its no issue :)

[digitalpoint]

Hmmm... still odd. What's the cache date on one of the pages in question?

[Design Agent]

Not sure, as it is a supplemental page and has no option to show cache from what I can see..

[digitalpoint]

You should be able to see supplemental cache just like normal with the cache: function.

[Design Agent]

cache:www.xxx.com/forums/private.php?do=newpm&u=24 - did not match any documents

hmm even odder.

[digitalpoint]

As I mentioned in PM, it's not supplemental... only a URL that Google knows about (but hasn't spidered). Can't have a cache if Google hasn't actually spidered it yet. :)

[Design Agent]

Thanks. Makes alot more sense then.