Is my robots.txt any good????

Discussion in 'robots.txt' started by WONGA, Dec 21, 2009.

0
  1. #1
    Hi guys after been repeatedly blocked by googles bot i would like some advice i got this one of here titled the best robots.txt for vbulletin yet i still was blocked so i have just changed it to this by adding / to the end of the Dissallow bits and removing the allow which he left on anyway can someone let me know thank you all:) Plus i have it as a txt file on my server is that OK??? **I am running vbulletin 3.8.4**

    This is the one i changed it to edited etc....
    User-agent: *
    Disallow: /sendmessage.php/
    Disallow: /newreply.php/
    Disallow: /profile.php/
    Disallow: /memberlist.php/
    Disallow: /editpost.php/
    Disallow: /moderator.php/
    Disallow: /online.php/
    Disallow: /poll.php/
    Disallow: /inlinemod.php/
    Disallow: /printthread.php/
    Disallow: /member.php/
    Disallow: /cgi-bin/
    Disallow: /login.php/
    Disallow: /private.php/
    Disallow: /report.php/
    Disallow: /threadrate.php/
    Disallow: /register.php/
    Disallow: /subscription.php/
    Disallow: /search.php/
    Disallow: /ajax.php/
    Disallow: /image.php/
    Disallow: /misc.php/
    Disallow: /newthread.php/
    Disallow: /attachment.php/
    Disallow: /reputation.php/
    Disallow: /global.php/
    Disallow: /showgroups.php/
    Disallow: /newattachment.php/
    Disallow: /cron.php/
    Disallow: /joinrequests.php/
    Disallow: /usernote.php/
    Disallow: /postings.php/
    Disallow: /calendar.php/
    Disallow: /usercp.php/

    The one i copied...
    User-Agent: *
    allow: /
    Disallow: /cgi-bin/
    Disallow: /ajax.php
    Disallow: /attachment.php
    Disallow: /calendar.php
    Disallow: /cron.php
    Disallow: /editpost.php
    Disallow: /global.php
    Disallow: /image.php
    Disallow: /inlinemod.php
    Disallow: /joinrequests.php
    Disallow: /login.php
    Disallow: /member.php
    Disallow: /memberlist.php
    Disallow: /misc.php
    Disallow: /moderator.php
    Disallow: /newattachment.php
    Disallow: /newreply.php
    Disallow: /newthread.php
    Disallow: /online.php
    Disallow: /poll.php
    Disallow: /postings.php
    Disallow: /printthread.php
    Disallow: /private.php
    Disallow: /profile.php
    Disallow: /register.php
    Disallow: /report.php
    Disallow: /reputation.php
    Disallow: /search.php
    Disallow: /sendmessage.php
    Disallow: /showgroups.php
    Disallow: /subscription.php
    Disallow: /threadrate.php
    Disallow: /usercp.php
    Disallow: /usernote.php
     
    Last edited: Dec 21, 2009
    WONGA, Dec 21, 2009 IP
  2. sevnrock

    sevnrock Peon

    Messages:
    241
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #2
    What do you mean "blocked by Googles bot"? By adding the / at the end of all your Disallow commands, you're allowing Google to index all the regular .php pages. You're ONLY blocking a url that looks like "www.mysite.com/sendmessage.php/" and allowing "www.mysite.com/sendmessage.php" which was the original purpose of blocking it.
     
    sevnrock, Dec 23, 2009 IP