Best robots.txt for vBulletin forums

robots.txt for vBulletin forums
I think this robots.txt suits for all vBulletin forums ..

If your forum in the main directory

if its in /forum

Also you can add these lines if you have a sitemap

Sitemap: http://www.YOURFORUM.com/sitemap.xml.gz

Code (markup):

Any suggestions ? and any improvements..please reply below

 

I'm pretty sure you need the sitemap to have the .xml file extension instead of the .gz extension. You want the file to be uncompressed and available to be read by others.

 

Google Supports that

 

You are correct. I guess I was worried about encountering problems if your webserver is not sending correct MIME headers for .gz files, some browsers may instead try open the sitemap.xml.gz file. Then all they will see is jibberish. But maybe this is a minor issue for only a few web servers out there.

 

very nice thanks for sharing

 

Thanks, never thought about doing that!

 

thank you for sharing man

 

After using the first part of this on my forum i didnt get indexed
so i changed it i took away the allow: / and also added / to the list of disallow is this correct im due to be indexed in next couple of days so any help would be apprieciated thanks.

Happy Holidays To All

 

this is the best robots text file IMHO for vBulletin.

 

i have it now n my site thank you mate you are great

 

Thank you Loco.M i used your version and works a treat

 

don't forget to change the sitemap line to reflect the location of your sitemap
or comment it out if you don't have a sitemap