1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt for WHMCS

Discussion in 'robots.txt' started by Stephen H.Cummings, Dec 26, 2012.

0
  1. #1
    Hello Everyone. I recently found a very useful page about robots.txt on WHMCS. Here's the detail:-

    How to allow all search engine spiders to index all files of Whmcs?


    Use the following content for your robots.txt file if you want to allow all search engine spiders to index all files of your Web site:


    User-agent: * 
Disallow:
    Code (markup):

    How to disallow all spiders to index any file of Whmcs?


    If you don't want search engines to index any file of your Web site, use the following:


    User-agent: * 
Disallow: /
    Code (markup):

    Do not know what to do? Copy and paste this code in a txt archive called "robots.txt" and upload it to Whmcs root directory:


    User-agent: * 
Allow: / 
User-agent: Mediapartners-Google 
Allow: / 
User-agent: Googlebot 
Allow: / 
User-agent: Adsbot-Google 
Allow: / 
User-agent: Googlebot-Image 
Allow: / 
User-agent: Googlebot-Mobile 
Allow: /
    Code (markup):
     
    Stephen H.Cummings, Dec 26, 2012 IP
  2. selectaupairs

    selectaupairs Greenhorn

    Messages:
    59
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    8
    #2
    If you are not allowed to crawler to some pages of your site you create robots.txt file for your site and create this type of code for this
    User-agent: * Disallow: /cgi-bin/
    Disallow: /http://www.spaculus.com/css/MenuMatic.css
    Disallow: /http://www.spaculus.com/software-product-development.as
    Disallow: /http://www.spaculus.com/b2b-b2c-solutio..
    Disallow: /http://www.spaculus.com/2
     
    selectaupairs, Dec 26, 2012 IP