What is the need of ROBOTS.TXT ?

Discussion in 'robots.txt' started by davindarrel, Dec 25, 2008.

0
  1. #1
    hi guys can anyone tell me whats is the need for this robots.txt ?

    i am a newbie and not much familiar abt such things so might asked some foolishness forgive me guys :(
     
    davindarrel, Dec 25, 2008 IP
  2. dcristo

    dcristo Illustrious Member

    Messages:
    19,776
    Likes Received:
    1,200
    Best Answers:
    7
    Trophy Points:
    470
    Articles:
    7
    #2
    dcristo, Dec 25, 2008 IP
  3. dmichael681

    dmichael681 Peon

    Messages:
    13
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    It is a file to give instructions about the site to the web robots. It is known as Robots Exclusion Protocol. But remember that it can't stop all the robots. Malware robots that scan the web for security vulnerabilities and email harvesters used by spammers will pay no attention. For more info visit robotstxt.org
     
    dmichael681, Dec 25, 2008 IP
  4. pageloadtime

    pageloadtime Peon

    Messages:
    9
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Here is info from google on how they use the robots txt file

    google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
     
    pageloadtime, Jan 19, 2009 IP
  5. manish.chauhan

    manish.chauhan Well-Known Member

    Messages:
    1,682
    Likes Received:
    35
    Best Answers:
    0
    Trophy Points:
    110
    #5
     
    manish.chauhan, Jan 20, 2009 IP