what is best situation to use robots.txt

Discussion in 'Search Engine Optimization' started by leadgeneration, Aug 18, 2011.

0
  1. #1
    I am little bit confuse regarding robots.txt .I want to know what is best condition to use this file, any body can explain to me .
     
    leadgeneration, Aug 18, 2011 IP
  2. russ.murray

    russ.murray Member

    Messages:
    127
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    26
    #2
    if you don't want to index your admin pages on Google then you can use it
     
    russ.murray, Aug 18, 2011 IP
  3. unknownpray

    unknownpray Active Member

    Messages:
    3,831
    Likes Received:
    14
    Best Answers:
    0
    Trophy Points:
    70
    #3
    Basically the purp0se of a robots.txt file is that you can prevent the Search Engine crawler from crawling these pages and getting them indexed, this will then prevent them from showing in the Search Engines search results. The pages that I would put in a robots.txt file would the printer friendly pages, the admin pages that shows the admin login and admin logout and any other page that I would not want to see in the results page.
     
    unknownpray, Aug 19, 2011 IP
  4. Alan Smith

    Alan Smith Active Member

    Messages:
    1,263
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    78
    #4
    If your site contain page like career.html and there's no need to have it indexed, so you can use the robots.txt file to tell the search engine spider to ignore it.
     
    Alan Smith, Aug 24, 2011 IP
  5. sisatel

    sisatel Active Member

    Messages:
    1,391
    Likes Received:
    10
    Best Answers:
    1
    Trophy Points:
    90
    #5
    At the time of onpage optimization, I am cresting the Robot.txt file. I agree with "unknownpray"'s statement.
     
    sisatel, Aug 24, 2011 IP