Robots.txt, Vbulletin & Google

Discussion in 'robots.txt' started by Design Agent, Nov 26, 2005.

0
  1. #1
    Are there any recommended pages that vbulletin creates that should be blocked from google ?

    For example - I have 'send pm' pages showing in my site: search. which are not necessary at all.

    Any other I should block specifically?
     
    Design Agent, Nov 26, 2005 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    What version of vBulletin are you using? vBulletin 3.5 does a really good job at using nofollow on stuff that shouldn't be indexed right out of the box.
     
    digitalpoint, Nov 26, 2005 IP
  3. Design Agent

    Design Agent Peon

    Messages:
    3,061
    Likes Received:
    154
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Version 3.5 (though I need to upgrade) to 3.51

    Pages like this are being indexed (I haven't checked to see which others yet)

    /forums/private.php?do=newpm&u=40
    /forums/private.php?do=newpm&u=24
    Which are coming up as supplemental and not necessary.
     
    Design Agent, Nov 26, 2005 IP
  4. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #4
    If they are supplemental, it's probably really old (before you upgraded to 3.5). I wouldn't worry about it since new ones won't get indexed and the old ones shouldn't get reindexed. But if it bugs you, just add private.php to your robots.txt exclude directives and get them out quick with this:

    http://services.google.com/urlconsole/controller
     
    digitalpoint, Nov 26, 2005 IP
  5. Design Agent

    Design Agent Peon

    Messages:
    3,061
    Likes Received:
    154
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Actually when I upgraded I moved the forum to a different address and used google's page removal tool for the old address to deindex all the pages first (kind of a mini wmw for that section), so its not that I guess.

    Thanks for the tips. I'll just leave it if its no issue :)
     
    Design Agent, Nov 26, 2005 IP
  6. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #6
    Hmmm... still odd. What's the cache date on one of the pages in question?
     
    digitalpoint, Nov 26, 2005 IP
  7. Design Agent

    Design Agent Peon

    Messages:
    3,061
    Likes Received:
    154
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Not sure, as it is a supplemental page and has no option to show cache from what I can see..
     
    Design Agent, Nov 26, 2005 IP
  8. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #8
    You should be able to see supplemental cache just like normal with the cache: function.
     
    digitalpoint, Nov 26, 2005 IP
  9. Design Agent

    Design Agent Peon

    Messages:
    3,061
    Likes Received:
    154
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Design Agent, Nov 26, 2005 IP
  10. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,334
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #10
    As I mentioned in PM, it's not supplemental... only a URL that Google knows about (but hasn't spidered). Can't have a cache if Google hasn't actually spidered it yet. :)
     
    digitalpoint, Nov 26, 2005 IP
  11. Design Agent

    Design Agent Peon

    Messages:
    3,061
    Likes Received:
    154
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Thanks. Makes alot more sense then.
     
    Design Agent, Nov 26, 2005 IP