Robots.txt

Discussion in 'All Other Search Engines' started by pavaniperumalla, Dec 2, 2010.

0
  1. #1
    Hi every body,


    http://www.website.com/robots.txt is my robots.txt file. Anyone can plz tel me is it right file?. What is my doubt is "User-agent: * " it is for all robots. what is the need to mension every robot?


    Thanks &Regards
    Pavani
     
    pavaniperumalla, Dec 2, 2010 IP
  2. gf111

    gf111 Peon

    Messages:
    42
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    hey .
    for what it is ?
     
    gf111, Dec 2, 2010 IP
  3. fastreplies

    fastreplies Banned

    Messages:
    2,336
    Likes Received:
    107
    Best Answers:
    0
    Trophy Points:
    190
    #3
    If he tells you, then he will have to kill you otherwise
    you will die at old age a genius. LOL

    :)

    fastreplies
     
    fastreplies, Dec 2, 2010 IP
  4. moneyideas

    moneyideas Member

    Messages:
    42
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #4
    Sure your file is fine.

    It allows all robots access to your whole site.
     
    moneyideas, Dec 3, 2010 IP
  5. zorbis

    zorbis Peon

    Messages:
    22
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Upload your Robots.txt file as below

    sitemap: Yourwebsite/sitemap.xml

    User-agent: *
    Allow: /
     
    zorbis, Nov 9, 2011 IP
  6. Alec Smith

    Alec Smith Peon

    Messages:
    6
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Sure your file is fine.

    It allows all robots access to your whole site.
     
    Alec Smith, Nov 10, 2011 IP
  7. googlesiterank

    googlesiterank Peon

    Messages:
    74
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #7
    Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
     
    googlesiterank, Nov 25, 2011 IP
  8. guusn

    guusn Active Member

    Messages:
    145
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    51
    #8
    If you use wordpress, you can choose KBROBOTS plugins
     
    guusn, Nov 25, 2011 IP
  9. SerVision

    SerVision Peon

    Messages:
    182
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #9
    The robots.txt file is used basically to allow all robots of all search engines like Google, Yahoo, bing etc to allow to crawl your site. Also if you think that there is any page in your site that the search engines should not crawl, you should also mention that page as well in this file.
     
    SerVision, Dec 6, 2011 IP
  10. superkev

    superkev Peon

    Messages:
    1
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    don't think yahoo use it anymore
     
    superkev, Dec 6, 2011 IP
  11. dpfocnehali

    dpfocnehali Greenhorn

    Messages:
    34
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    16
    #11
    Web site owners use the /robots.txt file to give instructions about their site to web robots this is called The Robots Exclusion Protocol.
    It works likes this:
    User-agent: *
    Disallow: /
    The "User-agent: *" means this section applies to all robots. The "Disallow: /" tells the robot that it should not visit any pages on the site.
     
    dpfocnehali, Dec 28, 2011 IP
  12. Smith Jones

    Smith Jones Peon

    Messages:
    110
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #12
    Hey guys,

    Robots.txt is basically used to tell the crawler where to crawl and which section you don't need to be crawled. While optimizing your site you can keep this text file under your root directory.

    Thanks and regards
    Smith Jones
     
    Smith Jones, Jul 31, 2012 IP