Robot.txt file

Discussion in 'Search Engine Optimization' started by Andrew80, Jul 30, 2011.

0
  1. #1
    Hey friends,

    Can anyone explain me about Robot.txt file... what this file does ?? what is the main purpose of this file in S.E.O. this question was asked me in my interview & i had no answer. please explain me about robot.txt
     
    Andrew80, Jul 30, 2011 IP
  2. hashif

    hashif Well-Known Member

    Messages:
    1,745
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    130
    #2
    The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable. Robots are often used by search engines to categorize and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites...

    Refer:http://en.wikipedia.org/wiki/Robots.txt

    Regards,
    Hashif
     
    hashif, Jul 30, 2011 IP
  3. Abhishek Kundu

    Abhishek Kundu Active Member

    Messages:
    133
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    58
    #3
    hi andrew80,

    in simple words robot.txt file is used to tell the search engine spider which directories you want them to visit and which to avoid. :)

    kindest,
    abhishek kundu
     
    Abhishek Kundu, Jul 30, 2011 IP
  4. Alan Smith

    Alan Smith Active Member

    Messages:
    1,263
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #4
    Most sites contain pages that should not be indexed by the search engines. Administrative pages, for example, ABC's "contact" page: "contact.html". There's no need to have it indexed, so we use the robots.txt file to tell the search engine spider (robot) to ignore it.
     
    Alan Smith, Aug 4, 2011 IP
  5. Bekky

    Bekky Peon

    Messages:
    183
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Robots.txt files helps us for google crawling. add in your Root URL. it helps us which page should be crawl or not.
     
    Bekky, Aug 5, 2011 IP
  6. sowellavena

    sowellavena Greenhorn

    Messages:
    83
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #6
    robot.txt is a file which provide instruction to robot that where to go and where not to go on your website.
    You can bound to crawler for not visiting on your any url.
     
    sowellavena, Aug 5, 2011 IP
  7. romysingh

    romysingh Peon

    Messages:
    14
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Hello Friend,
    Robots.txt file just simply tells the search engine bot to which file and which url is to index and which not... for complete reference about robots.txt visit to this link :_

    http://www.crunchyblogger.com/ultimate-robots-txt-for-wordpress/
     
    romysingh, Aug 5, 2011 IP
  8. Rajeev123

    Rajeev123 Peon

    Messages:
    100
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    Basically Robots.txt files helps  for google crawling and you can disallow any page and website..
     
    Rajeev123, Aug 5, 2011 IP
  9. jollydutta

    jollydutta Member

    Messages:
    79
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    36
    #9
    In one word robot.txt file is used to tell the search engine spider. It helps for google crawling.
     
    jollydutta, Aug 5, 2011 IP
  10. madinamadina

    madinamadina Peon

    Messages:
    52
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Robots.txt file is a simple notepad file, it contains the command like user agent: * means all search engine spider will crawl your files and folders and if you want to restrict the access to some files and folders than you have to specify the command like

    Disallow: /your file and folders and URL name.

    hope you have have understood now.
    and best of luck for the next interview
     
    madinamadina, Aug 5, 2011 IP