what is the impartance of ROBOT.TXT file

Discussion in 'Search Engine Optimization' started by Shefaligarg, Dec 1, 2010.

  1. #1
    If we don't include a ROBOT.TXT file, is there any disadvantages.
     
    Shefaligarg, Dec 1, 2010 IP
  2. Alan Smith

    Alan Smith Active Member

    Messages:
    1,263
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #2
    If you don't put robots.txt file then there is no any single disadvantage of that. But SEO point of view if you put that file then it is beneficial for you.
     
    Alan Smith, Dec 2, 2010 IP
  3. keyideas

    keyideas Peon

    Messages:
    90
    Likes Received:
    2
    Best Answers:
    0
    Trophy Points:
    0
    #3
    No any disadvantage but robots.txt file is used for restricting any directory of file which you don't want to crawl by the search engines.
     
    keyideas, Dec 2, 2010 IP
  4. Jeff Collision

    Jeff Collision Peon

    Messages:
    1,020
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #4
    Agree with Alan Smith. No issues if you doesn't insert your robots.txt in your site. It, only benefits you if you include it in your site as SEO point of view. Wish to share an article from SEO Complete Solution which provides the benefits of robots.txt & explains it very well. Go through it.
     
    Jeff Collision, Dec 2, 2010 IP
  5. seorupesh

    seorupesh Peon

    Messages:
    295
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    Websites that have undergone Search Engine Optimization may include robots exclusion standards, or robots.txt, which are text file protocols that control the search engine spiders. They advise the spiders to refrain from visiting parts of the Website that are either irrelevant to the site’s categorization or that may be private.
     
    seorupesh, Dec 2, 2010 IP
  6. Imdadc

    Imdadc Peon

    Messages:
    617
    Likes Received:
    10
    Best Answers:
    0
    Trophy Points:
    0
    #6
    That’s a great question. One of my blog I’ve a static robots.txt, so I’ve set up a dynamic example using code snippets from other sites: this robots.txt is what a user sees, and here is what various crawlers get on request of my robots.txt example.

    Of course crawlers don’t request a robots.txt file with a query string identifying themselves.

    Imdad
     
    Imdadc, Dec 2, 2010 IP
  7. sherone

    sherone Well-Known Member

    Messages:
    1,539
    Likes Received:
    16
    Best Answers:
    0
    Trophy Points:
    130
    #7
    It will help spiders to crawl all your pages. If you want, you can block the spiders through robots.txt
     
    sherone, Dec 2, 2010 IP
  8. imperiousseo

    imperiousseo Peon

    Messages:
    78
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #8
    If you dont want search engines to crawl any web page, then you can use robot.txt on that page code. There is no disadvantage from this code.
     
    imperiousseo, Dec 2, 2010 IP