Robots.txt For Vbulletin Forum

Discussion in 'robots.txt' started by Rezafcom, Sep 15, 2008.

  1. #1
    use this code for block crawler , not index admincp and etc.

    
    User-agent: *
    Disallow: /forums/ajax.php
    Disallow: /forums/attachment.php
    Disallow: /forums/calendar.php
    Disallow: /forums/cron.php
    Disallow: /forums/editpost.php
    Disallow: /forums/global.php
    Disallow: /forums/image.php
    Disallow: /forums/inlinemod.php
    Disallow: /forums/joinrequests.php
    Disallow: /forums/login.php
    Disallow: /forums/member.php
    Disallow: /forums/memberlist.php
    Disallow: /forums/misc.php
    Disallow: /forums/moderator.php
    Disallow: /forums/newattachment.php
    Disallow: /forums/newreply.php
    Disallow: /forums/newthread.php
    Disallow: /forums/online.php
    Disallow: /forums/poll.php
    Disallow: /forums/postings.php
    Disallow: /forums/printthread.php
    Disallow: /forums/private.php
    Disallow: /forums/profile.php
    Disallow: /forums/register.php
    Disallow: /forums/report.php
    Disallow: /forums/reputation.php
    Disallow: /forums/search.php
    Disallow: /forums/sendmessage.php
    Disallow: /forums/showgroups.php
    Disallow: /forums/subscription.php
    Disallow: /forums/threadrate.php
    Disallow: /forums/usercp.php
    Disallow: /forums/usernote.php
    
    Code (markup):
     
    Rezafcom, Sep 15, 2008 IP
  2. abdobasha2004

    abdobasha2004 Well-Known Member

    Messages:
    1,117
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    160
    #2
    why not leaving it to google
    u never know what is useful
    what is better to be crawled
    I really disallow some pages but i ll allow all page soon
    I see there is no value for this
     
    abdobasha2004, Nov 3, 2008 IP