Importance of Robots.txt file

Discussion in 'Search Engine Optimization' started by kenny_pure, Nov 25, 2008.

  1. #1
    guys

    does Robots.txt file affect your SE ranking.....

    i am not using Robots.txt file in one of my website....

    please tell me is there any harm?
     
    kenny_pure, Nov 25, 2008 IP
  2. Sarahsienna

    Sarahsienna Peon

    Messages:
    32
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Hi Friends in my opinion Robots.txt file will surely affect SE ranking because When a search engine crawler comes to your site, it will look for a special file on your site. That file is called robots.txt and it tells the search engine spider, which Web pages of your site should be indexed and which Web pages should be ignored.

    The robots.txt file is a simple text file (no HTML), that must be placed in your root directory



    Dear friend in short it will surely affect SE ranking.

    Regards

    Sarah
     
    Sarahsienna, Nov 25, 2008 IP
  3. marlonpowell

    marlonpowell Active Member

    Messages:
    464
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    85
    #3
    It is advisable to keep one. The robots.txt patterns are matched by simple substring comparisons. Care should be taken to make sure that patterns matching directories have the final '/' character appended, otherwise all files with names starting with that substring will match, rather than just those in the directory intended.


    source:wiki
     
    marlonpowell, Nov 25, 2008 IP
  4. kenny_pure

    kenny_pure Active Member

    Messages:
    604
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    58
    #4
    my i am running a site without robots.txt file and it is working very well on SEs
     
    kenny_pure, Nov 25, 2008 IP
  5. vickyseo

    vickyseo Banned

    Messages:
    712
    Likes Received:
    7
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Hi Friend,
    I think it's give some help As SERP point of view......
    Thanks!
     
    vickyseo, Nov 25, 2008 IP
  6. davejug1

    davejug1 Banned

    Messages:
    374
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    0
    #6
    Robots.txt does not affect your search rankings, Google (and other spiders) will find the links anyway. Robots.txt is used only to disallow certain robots from indexing your site or viewing pages with your site, as an example you would not want your "thank you" page indexed so you can block it from robots.txt

    Of course not all spiders are legitimate and will index your pages anyway, plus anyone who know about the robots file will search there first anyway so you will be showing these people what you don't want them to see.

    Kill robots.txt and add a meta tag to your headers instead
     
    davejug1, Nov 25, 2008 IP
  7. usamaung

    usamaung Banned

    Messages:
    215
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #7
    With robots.txt we can allow or disallow which file that should be crawl..so do not much eat the source of your server..and do not affect the SE
     
    usamaung, Nov 25, 2008 IP
  8. rena

    rena Peon

    Messages:
    1,987
    Likes Received:
    13
    Best Answers:
    0
    Trophy Points:
    0
    #8
    ITs not effecting any SE rank, this only to avoid crawl some pages,dir,file etc from Google crawl
     
    rena, Nov 25, 2008 IP
  9. gr8liverpoolfan

    gr8liverpoolfan Notable Member

    Messages:
    6,719
    Likes Received:
    538
    Best Answers:
    0
    Trophy Points:
    285
    #9
    In the earlier days, using robots.txt was one way of ensuring your site did not get the duplicate content penalty, especially if two or more pages were similar in most respects.

    If you want to disallow certain pages, use it.

    The only way it could affect your ranking would be for you to ask G,Y,MSN not to follow those pages which could possibly get you a rank ;)
     
    gr8liverpoolfan, Nov 25, 2008 IP
  10. RichardG87

    RichardG87 Peon

    Messages:
    134
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #10
    Although the robots.txt file is a very important file if you want to have a good ranking on search engines, many Web sites don't offer this file.
    If your Web site doesn't have a robots.txt file yet, read on to learn how to create one
     
    RichardG87, Nov 25, 2008 IP
  11. davejug1

    davejug1 Banned

    Messages:
    374
    Likes Received:
    17
    Best Answers:
    0
    Trophy Points:
    0
    #11
    Eh? Lol 0/3 nice one dude

    Want to see just how bad it can get?

    http://www.whitehouse.gov/robots.txt
    Code (markup):
     
    davejug1, Nov 25, 2008 IP
  12. prima

    prima Peon

    Messages:
    228
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #12
    robots.txt file prevents pages for search engine crawling.
     
    prima, Nov 25, 2008 IP
  13. prima

    prima Peon

    Messages:
    228
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #13
    robots.txt file prevents pages for search engine crawling.
     
    prima, Nov 25, 2008 IP
  14. xtreme fever

    xtreme fever Active Member

    Messages:
    472
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    95
    #14
    you can use a robots.txt file to list your sitemap. either sitemap.txt or sitemap.xml
     
    xtreme fever, Nov 25, 2008 IP
  15. tastro

    tastro Member

    Messages:
    68
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    41
    #15
    on default... all the pages on your site are allowed... you need robots only if you want to disallow the search engine to crawl the site.
     
    tastro, Nov 25, 2008 IP
  16. dota

    dota Banned

    Messages:
    756
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    0
    #16
    yes.google robots will give more effect with ur serp.
    more over your setting of robots texts
     
    dota, Nov 25, 2008 IP
  17. marlonpowell

    marlonpowell Active Member

    Messages:
    464
    Likes Received:
    8
    Best Answers:
    0
    Trophy Points:
    85
    #17

    Cool, that might be the longest robots.txt file!!!!
     
    marlonpowell, Nov 25, 2008 IP
  18. ShayneSherman

    ShayneSherman Peon

    Messages:
    93
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #18
    It will help with the duplicate penalty you can get for having/ not having www
     
    ShayneSherman, Nov 25, 2008 IP
  19. seoservicesvs

    seoservicesvs Member

    Messages:
    310
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #19
    Does not affect search rankings using of Robots.txt file. it is helps easy to crawling spider.
     
    seoservicesvs, Nov 25, 2008 IP
  20. NaughtyNeo

    NaughtyNeo Peon

    Messages:
    829
    Likes Received:
    41
    Best Answers:
    0
    Trophy Points:
    0
    #20
    As of now, robots.txt won't affect you ranking and you won't need one unless you want to block spiders from crawling your entire site.

    But it is advisable that you have at least a black robots.txt file uploaded to reduce the number of 404 hits caused by the spiders. Check your server stats.
    (This won't affect you if you are running a wordpress free hosted blog about Indian festival 2009. ;) )

    -Neo
     
    NaughtyNeo, Nov 26, 2008 IP