Do you use Robots.txt File? Here is the Importance.

Discussion in 'Search Engine Optimization' started by Rogi, Mar 14, 2009.

0
Page 2 of 2
  1. vl09

    vl09 Peon

    Messages:
    63
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #21
    Yes, robot.txt is very imp for site optimization and xml sitemap should be mentioned in it.
     
    vl09, Mar 17, 2009 IP
  2. newlogo

    newlogo Peon

    Messages:
    3,931
    Likes Received:
    11
    Best Answers:
    0
    Trophy Points:
    0
    #22
    yes especially useful when u not try to index some of pages in search engine
     
    newlogo, Mar 17, 2009 IP
  3. mmerlinn

    mmerlinn Prominent Member

    Messages:
    3,197
    Likes Received:
    819
    Best Answers:
    7
    Trophy Points:
    320
    #23
    Robots.txt is totally USELESS in my view. Why should I need a robots.txt when I can put a META tag in the TINY FRACTION of pages that I want to restrict faster and easier than in a separate robots.txt file????
     
    mmerlinn, Mar 20, 2009 IP
  4. zeekstern

    zeekstern Active Member

    Messages:
    872
    Likes Received:
    20
    Best Answers:
    0
    Trophy Points:
    60
    #24
    If your site has all the pages in one directory, then this might be true. What happens if you have a mysite.com/recommends folder or an mysite.com/images folder etc etc? Seems to me the robot.txt file would sure come in handy there;)
     
    zeekstern, Mar 21, 2009 IP
  5. bollywoodgeeks

    bollywoodgeeks Peon

    Messages:
    275
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #25
    I am surprised to see how many people are unaware of robots.txt file. This file is kept in the root directory and in fact we use it for all our sites… ;)
     
    bollywoodgeeks, Mar 21, 2009 IP
  6. mmerlinn

    mmerlinn Prominent Member

    Messages:
    3,197
    Likes Received:
    819
    Best Answers:
    7
    Trophy Points:
    320
    #26
    My site has over 4000 pages with more added as needed. There are around 200 subdirectories. There are only 3 files in each directory that need restriction, so doing a robots.txt file is way overkill for the few files that need restriction.
     
    mmerlinn, Mar 21, 2009 IP
  7. kiranvoleti

    kiranvoleti Peon

    Messages:
    40
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #27
    It is one of the key file for ecommers and corporate websites.
     
    kiranvoleti, Mar 22, 2009 IP
Page 2 of 2