Is this a good robots.txt for vbulletin?

Discussion in 'robots.txt' started by HomeRun, Mar 25, 2008.

  1. #1
    Is there something i disallowed u wouldn't disallow? is there something i missed to disallow

    and why?


    User-agent: *
    Disallow: /ajax.php
    Disallow: /ajax_cron.php
    Disallow: /attachment.php
    Disallow: /checkspelling.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /external.php
    Disallow: /faq.php
    Disallow: /global.php
    Disallow: /image.php
    Disallow: /joinrequest.php
    Disallow: /login.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newattatchment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /payment_gateway.php
    Disallow: /payments.php
    Disallow: /pdfthread.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /subcription.php
    Disallow: /subcriptions.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php

    Disallow: /admincp/
    Disallow: /images/
    Disallow: /modcp/
    Disallow: /clientscript/
    Disallow: /customavatars/
    Disallow: /customprofilepics/
    Disallow: /files/
    Disallow: /install/
    Disallow: /cpstyles/
    Disallow: /images/
    Disallow: /includes/
    Disallow: /subscriptions/
    Disallow: /attachments/

    Sitemap: http://www.xxx.com/sitemap_index.xml.gz
     
    HomeRun, Mar 25, 2008 IP
  2. essex

    essex Guest

    Messages:
    49
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Why would you disallow your FAQ ? To me thats an important part of your site.

    allow: /search.php
    allow: /poll.php
    ....etc
     
    essex, Mar 31, 2008 IP
  3. FightRice

    FightRice Peon

    Messages:
    1,082
    Likes Received:
    28
    Best Answers:
    0
    Trophy Points:
    0
    #3
    faq isn't an important part of 98% of vBulletin sites. Here is an vBulletin robots.txt thats pretty much the standard
     
    FightRice, Mar 31, 2008 IP