1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

googlebot, vbulletin, and robots.txt

Discussion in 'robots.txt' started by disgust, Aug 5, 2004.

0
  1. #1
    normally I don't even bother with robots.txt- I figure I can spare some bandwidth for some extra traffic once it gets into the search engines...

    but googlebot is really hammering my vbulletin- I'd like them to be able to view threads (and threads in the archive) fine, but the completely useless things that they keep on viewing I'd like to exclude. namely, the edit post, quote post, report a post to a moderator, etc.

    does anyone know of a robots.txt for vbulletin 2 that'll just keep them looking at the actual threads, and not the useless pages they can't do anything with?
     
    disgust, Aug 5, 2004 IP
  2. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #2
    I tried to do something similar when this forum first went up, but I ended up giving up and taking out the robots.txt file completely. The more content Google has, the better so I figure it's just one of those things with running a forum...
     
    digitalpoint, Aug 5, 2004 IP
  3. vprp

    vprp Peon

    Messages:
    274
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    here's what i have...

    User-agent: *
    Disallow: /arcade.php
    Disallow: /attachment.php
    Disallow: /avatar.php
    Disallow: /editpost.php
    Disallow: /member.php
    Disallow: /member2.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /private2.php
    Disallow: /report.php
    Disallow: /search.php
    Disallow: /sendtofriend.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /admincp/
    Disallow: /modcp/
    Disallow: /images/
    Disallow: /sendmessage.php
    Disallow: /register.php
    Disallow: /subscription.php
     
    vprp, Aug 5, 2004 IP
  4. disgust

    disgust Guest

    Messages:
    2,417
    Likes Received:
    133
    Best Answers:
    0
    Trophy Points:
    0
    #4
    was there any particular reason you gave up on it?

    vprp: thanks, I might try that :)
     
    disgust, Aug 5, 2004 IP
  5. Help Desk

    Help Desk Well-Known Member

    Messages:
    1,365
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    180
    #5
    More content is better serps and PR, so why bother?
     
    Help Desk, Aug 5, 2004 IP
  6. digitalpoint

    digitalpoint Overlord of no one Staff

    Messages:
    38,333
    Likes Received:
    2,613
    Best Answers:
    462
    Trophy Points:
    710
    Digital Goods:
    29
    #6
    Because bandwidth really isn't all that expensive, and I rank well in Google for pages other than thread and post pages as well...
     
    digitalpoint, Aug 5, 2004 IP