is robots.txt a good thing to have in forum root?

Discussion in 'robots.txt' started by icare, May 1, 2006.

  1. #1
    hello friends I am always confused... should I have robots.txt file in my root or not?
    can anyone tell me advantages and disadvantages of having it and not having it?
    For now i have following file in my root folder please let me know if there is anything wrong or should change?
    
        User-agent: *
        Disallow: /admincp/
        Disallow: /clientscript/
        Disallow: /cpstyles/
        Disallow: /customavatars/
        Disallow: /customprofilepics/
        Disallow: /images/
        Disallow: /modcp/
        Disallow: /ajax.php
        Disallow: /attachment.php
        Disallow: /calendar.php
        Disallow: /cron.php
        Disallow: /editpost.php
        Disallow: /global.php
        Disallow: /image.php
        Disallow: /inlinemod.php
        Disallow: /joinrequests.php
        Disallow: /login.php
        Disallow: /member.php
        Disallow: /memberlist.php
        Disallow: /misc.php
        Disallow: /moderator.php
        Disallow: /newattachment.php
        Disallow: /newreply.php
        Disallow: /newthread.php
        Disallow: /online.php
        Disallow: /poll.php
        Disallow: /postings.php
        Disallow: /printthread.php
        Disallow: /private.php
        Disallow: /profile.php
        Disallow: /register.php
        Disallow: /report.php
        Disallow: /reputation.php
        Disallow: /search.php
        Disallow: /sendmessage.php
        Disallow: /showgroups.php
        Disallow: /subscription.php
        Disallow: /threadrate.php
        Disallow: /usercp.php
        Disallow: /usernote.php
        Disallow: /faq.php
    
    
    
    
    
    
    
    
    PHP:

     
    icare, May 1, 2006 IP
  2. mhdoc

    mhdoc Tauren

    Messages:
    840
    Likes Received:
    33
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Have you seen the suggested robots.txt forum article at websitepublisher.net? It's in the vBulletin article link on the home page.
     
    mhdoc, May 1, 2006 IP
  3. ash1

    ash1 Active Member

    Messages:
    2,240
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    88
    #3
    robots.txt is only good if you are hiding folders from spiders imho, but im no expert,
     
    ash1, May 1, 2006 IP
  4. elcash

    elcash Peon

    Messages:
    131
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #4
    I use my robots.txt in exactly the same way, to give recommendations to spiders on pages that they don't need to crawl. Remember that bots can choose to ignore robots.txt, so for sensitive content that *needs* to be protected, you should probably put a directive in .htaccess...
     
    elcash, May 5, 2006 IP