1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

Purpose of robots.txt

Discussion in 'robots.txt' started by umairraja52, Mar 16, 2011.

0
  1. #1
    I want to know, what is the purpose of robots.txt file in websites, how we can use it and can get benefits of it.
     
    umairraja52, Mar 16, 2011 IP
  2. dreteh

    dreteh Member

    Messages:
    514
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    33
    #2
    dreteh, Mar 16, 2011 IP
  3. Alan Smith

    Alan Smith Active Member

    Messages:
    1,263
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #3
    Robots.txt helps the crawler to know what to crawl and what not to.
     
    Alan Smith, Mar 18, 2011 IP
  4. iampapan

    iampapan Active Member

    Messages:
    111
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    73
    #4
    Thanks for sharing this information. It will help me at interview board.
     
    iampapan, Mar 22, 2011 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
    Robots.txt is basically used to control search engines crawlers or any other crawlers to visit private sections of your website that you do not want to show them.
     
    manish.chauhan, Mar 22, 2011 IP
  6. vinoth.t

    vinoth.t Peon

    Messages:
    156
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #6
    From robot.txt file you can specify the crawlers about accessing your website. Like which folder/file should be get index and more. This article will be helpful for you. Create Robot.txt File
     
    vinoth.t, Mar 23, 2011 IP
  7. techntuts

    techntuts Member

    Messages:
    197
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #7
    robots.txt file helps the search engine bots to crawl your website and helps in indexing your websites easily. There are many bad robots also it is recommended to use robots.txt and mention the folders to disallow from indexing and crawling.
     
    techntuts, Mar 26, 2011 IP
  8. drewtoby

    drewtoby Well-Known Member

    Messages:
    763
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    108
    #8
    Thanks, I was wondering what its purpose was too!
     
    drewtoby, Mar 26, 2011 IP
  9. Brunzig

    Brunzig Peon

    Messages:
    55
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Thanks to share that information with us.
     
    Brunzig, Apr 5, 2011 IP
  10. Oliveras11

    Oliveras11 Greenhorn

    Messages:
    17
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    11
    #10
    Robots.txt file is used to tell the search engine which files and folders to be crawled/indexed and which should not be crawled/indexed.always Place the robots.txt file on the root folder/level. :)
     
    Last edited: Apr 7, 2011
    Oliveras11, Apr 7, 2011 IP