Hello Everyone. I recently found a very useful page about robots.txt on WHMCS. Here's the detail:- How to allow all search engine spiders to index all files of Whmcs? Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site: User-agent: * Disallow: Code (markup): How to disallow all spiders to index any file of Whmcs? If you don't want search engines to index any file of your Web site, use the following: User-agent: * Disallow: / Code (markup): Do not know what to do? Copy and paste this code in a txt archive called "robots.txt" and upload it to Whmcs root directory: User-agent: * Allow: / User-agent: Mediapartners-Google Allow: / User-agent: Googlebot Allow: / User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Mobile Allow: / Code (markup):
If you are not allowed to crawler to some pages of your site you create robots.txt file for your site and create this type of code for this User-agent: * Disallow: /cgi-bin/ Disallow: /http://www.spaculus.com/css/MenuMatic.css Disallow: /http://www.spaculus.com/software-product-development.as Disallow: /http://www.spaculus.com/b2b-b2c-solutio.. Disallow: /http://www.spaculus.com/2