robots.txt - each directory need a robots.txt?

Discussion in 'Search Engine Optimization' started by nameonthecake, Aug 3, 2007.

0
  1. #1
    Generally you put your robots.txt file in the root directory.

    Can I place one in a directory besides the root.

    For example:
    www.mywebsite.com/forum/

    Could I place one in the forum directory without any problems?
     
    nameonthecake, Aug 3, 2007 IP
  2. jajed

    jajed Peon

    Messages:
    75
    Likes Received:
    4
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Spiders only check for robots.txt in the root directory, anywhere else is useless.
     
    jajed, Aug 3, 2007 IP
  3. nameonthecake

    nameonthecake Banned

    Messages:
    336
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #3
    ok thanks...
     
    nameonthecake, Aug 3, 2007 IP
  4. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #4
    you must add the urls that you dont want bots to follow with /forum/ folder to the root of your site
     
    trichnosis, Aug 4, 2007 IP
  5. nameonthecake

    nameonthecake Banned

    Messages:
    336
    Likes Received:
    1
    Best Answers:
    0
    Trophy Points:
    0
    #5
    ya i remember now...

    thanks
     
    nameonthecake, Aug 4, 2007 IP