what to include in robot.txt and what to not.

Discussion in 'Search Engine Optimization' started by uzma, Dec 30, 2009.

  1. #1
    I want to create a robot.txt file for my dynamic website. I was wondering which files/directories to include and which to exclude. Is there any free tool which could help me in generating robot.txt file.
     
    uzma, Dec 30, 2009 IP
  2. Canonical

    Canonical Well-Known Member

    Messages:
    2,223
    Likes Received:
    141
    Best Answers:
    0
    Trophy Points:
    110
    #2
    Every site's requirements are different. You only need a robots.txt file if you have specific files/folders that you don't want crawled. If you have no clue what should not be crawled then I would suggest you continue to live without one. They are not required. You can do more harm than good.
     
    Canonical, Dec 30, 2009 IP
  3. sevnrock

    sevnrock Peon

    Messages:
    241
    Likes Received:
    3
    Best Answers:
    0
    Trophy Points:
    0
    #3
    I agree with Canonical. It depends on your website. Usually bigger and more complex websites should use a robots.txt to ensure only the correct pages get crawled / indexed (eg. sites with shopping carts). But if your site is small and only contains a few static pages - you probably don't need one (eg. portfolio sites).

    If you want a (sort of) detailed guide on how to write one. Check here: http://www.webmasterindex.net/seo-how-to-write-a-robots-txt-file/
     
    sevnrock, Dec 30, 2009 IP
  4. HOD-Jardin

    HOD-Jardin Peon

    Messages:
    38
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #4
    HOD-Jardin, Dec 30, 2009 IP
  5. uzma

    uzma Peon

    Messages:
    44
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #5
    thank you guys ...
     
    uzma, Dec 31, 2009 IP