Robot.txt

Discussion in 'Search Engine Optimization' started by roarsin, Apr 11, 2011.

0
  1. #1
    Dear all respected members i want to know that what is robot.txt file ? and how it works ? and what is it importance for SEO and Search Engines


    Thank you
     
    roarsin, Apr 11, 2011 IP
  2. Fadi4u

    Fadi4u Active Member

    Messages:
    135
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    71
    #2
    Robot.txt is the file that Allows crawlers to crawl your website.. You can also restrict pages not to be crawled by putting them in robot.txt
     
    Fadi4u, Apr 11, 2011 IP
  3. qast123

    qast123 Peon

    Messages:
    29
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    Robot.txt is a file you put on your site to tell search robots which pages you would like not to visit.
     
    qast123, Apr 11, 2011 IP
  4. tarunyadavseo

    tarunyadavseo Member

    Messages:
    107
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #4
    Robot.txt file is used for a crawling your website.
     
    tarunyadavseo, Apr 11, 2011 IP
  5. bownsmith

    bownsmith Peon

    Messages:
    88
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    bownsmith, Apr 11, 2011 IP
  6. jvfconsulting

    jvfconsulting Active Member

    Messages:
    1,089
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    90
    #6
    jvfconsulting, Apr 11, 2011 IP
  7. chandrapal

    chandrapal Peon

    Messages:
    60
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Web Robots (also known as Web Wanderers, Crawlers, or Spiders), are programs that traverse the Web automatically. Search engines such as Google use them to index the web content, spammers use them to scan for email addresses, and they have many other uses.
     
    chandrapal, Apr 11, 2011 IP
  8. rt.maverick007

    rt.maverick007 Active Member

    Messages:
    238
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    73
    #8
    Good answer.... me too didn't know why we use robots.txt bt now i can understand..
     
    rt.maverick007, Apr 11, 2011 IP
  9. jayrudalf

    jayrudalf Peon

    Messages:
    347
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    Robot.txt use to let search engines crawlers know that the particular page or link or image need to crawl or index or not.
     
    jayrudalf, Apr 11, 2011 IP
  10. ayushi infotech

    ayushi infotech Peon

    Messages:
    1,814
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #10
    nice link about robot.txt nice explanation
     
    ayushi infotech, Apr 12, 2011 IP
  11. mr_clicks

    mr_clicks Greenhorn

    Messages:
    70
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #11
    The robots.txt file should be used to either block the crawlers from accessing a site or certain pages within the site or simply informing the crawlers that they can have full access to a website. I always think it is good practice to incorporate the address of your sitemaps within the robots.txt file to look to enhance the indexing of your site further.
     
    mr_clicks, Apr 12, 2011 IP