Robots.txt for Vbulletin

I have a forum on Vbulletin that does not have any Robots.txt file.
It has VBSEO running, so the URL's are all rewritten.

Does anyone know how to setup robots.txt for such a forum?

 

I think you can start by disallowing the unnecessary folders:

User-agent: *
Disallow: /admincp/
Disallow: /modcp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customcode/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /includes/
Disallow: /signaturepics/

Code (markup):

 

Hey !!

Thanks for the that .. glad someone here sent me a sample.
thanks again.

 

Do you think i should disallow all php files?
like showthread.php , because all the urls are rewritten by vbseo ?

 

If you feel that those files getting indexed would hurt you, go ahead. Personally, I'd leave the individual files alone.

 

Thats not actually the point Chewbacca. I think that if i leave the indiviual files alone, google may index them, this way i will have two indexed pages. And to my knowledge google does not like duplicate content

 

I know that, but won't Googlebot get 301s when it tries to index those files?

 

Have you checked with vbSEO's forum? I've never been a big fan of vbSEO but I'd say they'd have a good template to go off of if needed.

 

Im not sure about that , seem as if i will have to look into this...~~

 

Thanks,
I'll certainly go for that.

 

thanks for this. i was looking for a vbulletin robots.txt file

 

If anyone is still looking for this here is a good standard vbulletin 4x robots.txt file / code:

User-agent: *
Allow: /
Disallow: /admincp/
Disallow: /modcp/
Disallow: /clientscript/
Disallow: /cpstyles/
Disallow: /customavatars/
Disallow: /customcode/
Disallow: /customprofilepics/
Disallow: /images/
Disallow: /includes/
Disallow: /signaturepics/

Code (markup):