1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Allowing online seo analytics tools to crawl my website?

Discussion in 'Search Engine Optimization' started by janiman, Oct 24, 2015.

0
  1. #1
    I have disallowed all the bots to crawl my website because it is under construction.
    What I want is just to allow websites like woorank.com and similar online seo analytics tools to crawl my website.
    So what changes should I make to my robots.txt, so these seo analytics tools can crawl it.
    The website has some important company data they don't want anyone to access, that's why i can't allow all.

    Error
    This URL cannot be reviewed. There may be several reasons for this: the URL does not exist, the website does not allow WooRank to make reviews, the website denies both our HEAD and GET requests, the DNS sends to an infinite loop or the robots.txt page does not allow access to our bot.
     
    janiman, Oct 24, 2015 IP
  2. redesignunit@gmail.com

    redesignunit@gmail.com Banned

    Messages:
    230
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    108
    #2
    you need to ad exception rule for blocking several bot and that's it.
     
    redesignunit@gmail.com, Oct 25, 2015 IP