what is robot.txt?

Discussion in 'Guidelines / Compliance' started by kashadvertise, Aug 21, 2007.

0
  1. #1
    I dont know about this thing and now i want to know.
    Please tell me abt robot.txt

    what is it and how it works
     
    kashadvertise, Aug 21, 2007 IP
  2. jmhyer123

    jmhyer123 Peon

    Messages:
    542
    Likes Received:
    25
    Best Answers:
    0
    Trophy Points:
    0
    #2
    it is a special file that tells search engines what they can and can't index and include in search results. It is basically a file that holds instructions for search engine spiders.
     
    jmhyer123, Aug 21, 2007 IP
  3. Adpubster

    Adpubster Peon

    Messages:
    4,017
    Likes Received:
    153
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Adpubster, Aug 21, 2007 IP
  4. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #4
    pls visit robotstxt.org to learn more about robots.txt
     
    trichnosis, Aug 21, 2007 IP
  5. Dwaighty

    Dwaighty Peon

    Messages:
    358
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #5
    You only need a robots.txt file if you have pages that you wouldn't like search engines to index. The lack of a robots.txt file doesn't affect your SERPS and search engines will index your website normally.
     
    Dwaighty, Aug 22, 2007 IP
  6. Adpubster

    Adpubster Peon

    Messages:
    4,017
    Likes Received:
    153
    Best Answers:
    0
    Trophy Points:
    0
    #6
    You can now place a reference to your sitemap in a robots.txt and more and more of the bots are starting to recognize this.
     
    Adpubster, Aug 22, 2007 IP