hello friends I am always confused... should I have robots.txt file in my root or not? can anyone tell me advantages and disadvantages of having it and not having it? For now i have following file in my root folder please let me know if there is anything wrong or should change? User-agent: * Disallow: /admincp/ Disallow: /clientscript/ Disallow: /cpstyles/ Disallow: /customavatars/ Disallow: /customprofilepics/ Disallow: /images/ Disallow: /modcp/ Disallow: /ajax.php Disallow: /attachment.php Disallow: /calendar.php Disallow: /cron.php Disallow: /editpost.php Disallow: /global.php Disallow: /image.php Disallow: /inlinemod.php Disallow: /joinrequests.php Disallow: /login.php Disallow: /member.php Disallow: /memberlist.php Disallow: /misc.php Disallow: /moderator.php Disallow: /newattachment.php Disallow: /newreply.php Disallow: /newthread.php Disallow: /online.php Disallow: /poll.php Disallow: /postings.php Disallow: /printthread.php Disallow: /private.php Disallow: /profile.php Disallow: /register.php Disallow: /report.php Disallow: /reputation.php Disallow: /search.php Disallow: /sendmessage.php Disallow: /showgroups.php Disallow: /subscription.php Disallow: /threadrate.php Disallow: /usercp.php Disallow: /usernote.php Disallow: /faq.php PHP:
Have you seen the suggested robots.txt forum article at websitepublisher.net? It's in the vBulletin article link on the home page.
I use my robots.txt in exactly the same way, to give recommendations to spiders on pages that they don't need to crawl. Remember that bots can choose to ignore robots.txt, so for sensitive content that *needs* to be protected, you should probably put a directive in .htaccess...