1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Help with Robots.txt

Discussion in 'robots.txt' started by infopages, Sep 29, 2017.

0
  1. #1
    I have created website which is related to visa consultancy but didnt created robots.txt and then created one with minimal information on it.
    User-agent: *
    Disallow: /

    Can anyone help me to get right file with correct or better information in this.
    Thank you in advance
     
    infopages, Sep 29, 2017 IP
  2. qwikad.com

    qwikad.com Illustrious Member Affiliate Manager

    Messages:
    7,151
    Likes Received:
    1,656
    Best Answers:
    29
    Trophy Points:
    475
    #2
    Your current file is disallowing everything. Is that what you really want? You do not want your site to be indexed at all? You can find a complete info on what to put in your robots.txt file here:

    http://tools.seobook.com/robots-txt/
     
    qwikad.com, Sep 29, 2017 IP
  3. Robin Shukar

    Robin Shukar Peon

    Messages:
    7
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    1
    #3
    User-agent: *
    Disallow:
    you can also add your sitemap here.....
     
    Robin Shukar, Nov 21, 2017 IP
  4. Sumit_Singh

    Sumit_Singh Well-Known Member

    Messages:
    716
    Likes Received:
    64
    Best Answers:
    6
    Trophy Points:
    100
    #4
    Currently, the code you have written in your robots.txt file is disallowing all Search Engine Bots to crawling and indexing your website. If you are looking for the same then it is correct but if you want to robots text in your website and do not want to disallow any bot or page for the crawling, then go with the code that Robin wrote above.
    And it is not compulsory to add sitemap.xml file in the robots.txt file.
     
    Sumit_Singh, Nov 21, 2017 IP
  5. Elevate

    Elevate Greenhorn

    Messages:
    8
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #5
    As other people stated, you are blocking off everything from being crawled. Without seeing the files you have on your server, I would suggest blocking any admin folder at a minimum. Could you post a bit more information so we can help you out?
     
    Elevate, Aug 3, 2018 IP