What are the main uses in robots.txt

Discussion in 'Search Engine Optimization' started by balarachapally, Sep 29, 2014.

0
  1. #1
    What are the main uses in robots.txt. Could you please explain me all commands in robots.txt file?
     
    balarachapally, Sep 29, 2014 IP
  2. Siservices

    Siservices Active Member

    Messages:
    129
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    53
    #2
    Robots.txt is just a file where you tell Search Engine Bots which page to to allow and disallow for crawling. But if you don't want to disallow any search engine then you can leave it blank or simple way is don't upload(But that is not recommended). You can know everything in detail from here - Type this - robotstxt.org/robotstxt.html(Just avoided a link on post).
     
    Siservices, Sep 29, 2014 IP
  3. behnamy

    behnamy Greenhorn

    Messages:
    38
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    20
    #3
    and also you can give the address of your sitemap in robot.txt
     
    behnamy, Sep 30, 2014 IP
  4. Seo Power

    Seo Power Greenhorn

    Messages:
    48
    Likes Received:
    2
    Best Answers:
    2
    Trophy Points:
    18
    #4
    A robots.txt file is a file that prevents web crawlers like googlebot from crawling certain pages of your website. For example, if you disallow /contact-us in your robots.txt file, the url...

    www.yourdomain.com/contact-us

    ...will not appear in Google's index.

    Here's an article from Google that explains more about the uses of robots.txt files and how to use them: https://support.google.com/webmasters/answer/6062608?hl=en
     
    Seo Power, Oct 1, 2014 IP
  5. maddenitrous

    maddenitrous Well-Known Member

    Messages:
    190
    Likes Received:
    3
    Best Answers:
    1
    Trophy Points:
    118
    #5
    Mainly robots.txt file is used to deindex pages of your website, or completely deindex your website from search engines. It is useful to use when some of your website pages no longer exist and you want to deindex them from search engines.
     
    maddenitrous, Oct 2, 2014 IP
  6. Fix Johnson

    Fix Johnson Well-Known Member

    Messages:
    238
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    103
    #6
    So if I put a list of de-indexed pages in Robots.txt file it will work for all bots, not just Google?
     
    Fix Johnson, Oct 2, 2014 IP
  7. behnamy

    behnamy Greenhorn

    Messages:
    38
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    20
    #7
    you must search for the name of other search engines bot and put the name in robots.txt, for example the name of google crawler bot is "googlebot" and you can disallow googlebot to index your websites and allow others to index...
     
    behnamy, Oct 2, 2014 IP
  8. devandkumar11

    devandkumar11 Member

    Messages:
    76
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #8
    Robots.txt is a file to tell Search Engine's bot to which page need to crawl or which is not? In a single line, To Control Search Engine's Bot activity on your website, you have to create Robots.txt file.
     
    devandkumar11, Oct 7, 2014 IP
  9. Eliza Anderson

    Eliza Anderson Member

    Messages:
    69
    Likes Received:
    4
    Best Answers:
    3
    Trophy Points:
    43
    #9
    If I explain it to you in a simple way, it's main purpose is to tell the search engine's crawler whether or not to crawl a particular page.
     
    Eliza Anderson, Oct 8, 2014 IP
  10. patco

    patco Well-Known Member

    Messages:
    2,035
    Likes Received:
    47
    Best Answers:
    17
    Trophy Points:
    100
    #10
    You can tell Search Engines what to visit on your website and what not! For a blog, don't use it, but for a website for example written in PHP, robots. txt might take place! :)
     
    patco, Oct 8, 2014 IP
  11. umairhp

    umairhp Banned

    Messages:
    51
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    48
    #11
    If you do not want the Search engines to crawl specific pages on your website, Admin pages for example then you can do it through robots.txt
     
    umairhp, Oct 11, 2014 IP
  12. Jorge35

    Jorge35 Member

    Messages:
    24
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    31
    #12
    robots.txt allows search engines to crawl or un-crawl the links that you given in your website. It has ti be done with "rel" attribute as "dofollow" and "nodofollow" here you can see "dofollow" let search engines allowed to the crawl the link and "nodofollow" does not allowed to crawl it what have you given. It is used for credentials usage like bank transaction etc..,
     
    Jorge35, Oct 15, 2014 IP