Hello DP Members, I have a query. My website is a CMS and all the pages are coming from the database. Although these all pages are static in nature. But the Website is developed in Content Management System. So my question is, how to create robots.txt for such a website? Is it ok to simply disallow all the unnecessary folders to be crawled. Or should I add something more to make my pages crawled by the Bots and Spiders. Your Response is much awaited. Thanks in advance, Regards