what is the best robots.txt file for vbulletin 4? to prevent duplicate content? I serched a lt and find nothing !
if you don't want duplicate content block post.php User-agent: * Disallow: *.js Disallow: /clientscript/ Disallow: /cpstyles/ Disallow: /customavatars/ Disallow: /customprofilepics/ Disallow: /images/ Disallow: /ajax.php Disallow: /attachment.php Disallow: /calendar.php Disallow: /cron.php Disallow: /editpost.php Disallow: /global.php Disallow: /image.php Disallow: /inlinemod.php Disallow: /joinrequests.php Disallow: /login.php Disallow: /member.php Disallow: /memberlist.php Disallow: /misc.php Disallow: /moderator.php Disallow: /newattachment.php Disallow: /newreply.php Disallow: /newthread.php Disallow: /online.php Disallow: /poll.php Disallow: /postings.php Disallow: /printthread.php Disallow: /private.php Disallow: /profile.php Disallow: /register.php Disallow: /report.php Disallow: /reputation.php Disallow: /search.php Disallow: /sendmessage.php Disallow: /showpost.php Disallow: /threadrate.php Disallow: /usercp.php Disallow: /usernote.php #Finally, list the path to your sitemap: Sitemap: http://www.gamehaxerz.com/sitemap_index.xml.gz
This is a very good sitemap. Thanks! In my robots.txt I also have: Disallow: /showgroups.php Disallow: /subscription.php Should I disallow also all those files like: /admincp/, /signaturepics/, /customavatars/... etc. ?