How to work robot.txt file.?

Discussion in 'Search Engine Optimization' started by offsitenoc, Jul 21, 2010.

0
  1. #1
    Please tell me How to work robot.txt file.?
     
    offsitenoc, Jul 21, 2010 IP
  2. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #2
    manish.chauhan, Jul 21, 2010 IP
  3. apolosgreen

    apolosgreen Peon

    Messages:
    22
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Robots.txt file is just an easy text file store in the web server's root index. It t tells the web spiders not to crawl the web pages of the site.
     
    apolosgreen, Jul 21, 2010 IP
  4. Prakashets

    Prakashets Active Member

    Messages:
    106
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    58
    #4
    Robots.txt is a spider protocol file which is allows search engine which Web pages of your site should be indexed and which Web pages should be ignored
     
    Prakashets, Jul 21, 2010 IP
  5. Alan Smith

    Alan Smith Active Member

    Messages:
    1,263
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #5
    Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it.
    http://www.robotstxt.org/robotstxt.html
    Here given link will help you a lot, All the information regarding robots.txt is available.
     
    Alan Smith, Sep 13, 2010 IP