Is it necessary to upload a robots.txt file to the website?

Discussion in 'Search Engine Optimization' started by Anjali Gupta, Aug 23, 2013.

0
  1. #1
    I have a new website, Can anyone suggest that is there compulsion to upload a robots.txt file to my website, If i don't want to disallow any page?
     
    Anjali Gupta, Aug 23, 2013 IP
  2. royalstar_23

    royalstar_23 Well-Known Member

    Messages:
    265
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    140
    Articles:
    4
    #2
    Well, Its not a compulsion..But it guides search engine as which directory he should not view, For example in our website we have config folder, Log folder, Temp folder.... I would like to google to crawl that too from my website so i would preferably direct the google/search engines to leave those directories from the website..

    Hope my answers is clear to you.. Let me know if you need any assistance.

    Regards
    Vismit
     
    royalstar_23, Aug 23, 2013 IP
  3. Kianu42

    Kianu42 Member

    Messages:
    510
    Likes Received:
    15
    Best Answers:
    5
    Trophy Points:
    43
    #3
    Uploading robot.txt file is a must have for a better crawlability of your site and thus improving its onpage optimization.
    There is a plenty of free robot.txt generators and there is one in GWT.
     
    Kianu42, Aug 23, 2013 IP