Robots.txt

Discussion in 'Google' started by mellisajohn, Sep 24, 2009.

0
  1. #1
    Hi,
    What is the advantage of using robots.txt file?
    Is it necessary every site needs to have a robots.txt file?

    Regards
    Mellisa
     
    mellisajohn, Sep 24, 2009 IP
  2. goldfish

    goldfish Active Member

    Messages:
    408
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    78
    #2
    robot.txt to restrict the search engine bot to crawl your site or any url. Robot. txt will give the instruction to search engine bot to crawl only this section.... like that
     
    goldfish, Sep 24, 2009 IP
  3. seocornamedia

    seocornamedia Guest

    Messages:
    42
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    robots.txt is important, b'coz of it tell to SE crawler to crawl the websites.
    if you have not crawl some webpages to the crawl then u can do via robots.txt.....:)
     
    seocornamedia, Sep 24, 2009 IP