1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Best robots.txt for vBulletin forums

Discussion in 'robots.txt' started by remember123, Aug 22, 2009.

  1. #1
    robots.txt for vBulletin forums
    I think this robots.txt suits for all vBulletin forums ..

    If your forum in the main directory
    if its in /forum
    Also you can add these lines if you have a sitemap
    Sitemap: http://www.YOURFORUM.com/sitemap.xml.gz
    Code (markup):
    Any suggestions ? and any improvements..please reply below ;)
    SEMrush
     
    remember123, Aug 22, 2009 IP
    SEMrush
  2. awesometbn

    awesometbn Peon

    Messages:
    268
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #2
    I'm pretty sure you need the sitemap to have the .xml file extension instead of the .gz extension. You want the file to be uncompressed and available to be read by others.
     
    awesometbn, Sep 9, 2009 IP
  3. remember123

    remember123 Peon

    Messages:
    620
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Google Supports that :D
     
    remember123, Sep 13, 2009 IP
  4. awesometbn

    awesometbn Peon

    Messages:
    268
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    0
    #4
    You are correct. I guess I was worried about encountering problems if your webserver is not sending correct MIME headers for .gz files, some browsers may instead try open the sitemap.xml.gz file. Then all they will see is jibberish. But maybe this is a minor issue for only a few web servers out there.
     
    awesometbn, Sep 13, 2009 IP
  5. hafizahmedraza

    hafizahmedraza Active Member

    Messages:
    142
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #5
    very nice thanks for sharing
     
    hafizahmedraza, Dec 17, 2009 IP
  6. Crazace2006

    Crazace2006 Active Member

    Messages:
    143
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #6
    Thanks, never thought about doing that!
     
    Crazace2006, Dec 19, 2009 IP
  7. mamiman

    mamiman Peon

    Messages:
    21
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    thank you for sharing man
     
    mamiman, Dec 21, 2009 IP
  8. WONGA

    WONGA Active Member

    Messages:
    89
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    50
    #8
    After using the first part of this on my forum i didnt get indexed:mad:
    so i changed it i took away the allow: / and also added / to the list of disallow is this correct im due to be indexed in next couple of days so any help would be apprieciated thanks.

    Happy Holidays To All:D
     
    WONGA, Dec 23, 2009 IP
  9. Brandon Sheley

    Brandon Sheley Illustrious Member

    Messages:
    9,720
    Likes Received:
    611
    Best Answers:
    2
    Trophy Points:
    420
    #9
    this is the best robots text file IMHO for vBulletin.
     
    Brandon Sheley, Dec 23, 2009 IP
  10. bond010007

    bond010007 Peon

    Messages:
    4
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    i have it now n my site thank you mate you are great
     
    bond010007, Jan 3, 2010 IP
  11. WONGA

    WONGA Active Member

    Messages:
    89
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    50
    #11
    Thank you Loco.M i used your version and works a treat:D
     
    WONGA, Jan 8, 2010 IP
    Brandon Sheley likes this.
  12. Brandon Sheley

    Brandon Sheley Illustrious Member

    Messages:
    9,720
    Likes Received:
    611
    Best Answers:
    2
    Trophy Points:
    420
    #12
    don't forget to change the sitemap line to reflect the location of your sitemap :)
    or comment it out if you don't have a sitemap
     
    Brandon Sheley, Jan 8, 2010 IP