Robots.txt please help!!!

Discussion in 'robots.txt' started by frio, Jul 30, 2007.

0
  1. #1
    Hi guys, i have a problem with our site's robots.txt.
    I want to exclude a folder but it cannot be excluded because its not really a physical directory. it's a sort of rewrite URL.

    I've created a <Meta name = "robots" content="noindex, nofollow"> on the page.. will this thing help??? but I'm not able to add the robots.txt file on the folder:(

    Please help me guys!
    thanks in advance :D
     
    frio, Jul 30, 2007 IP
  2. vishwaa

    vishwaa Well-Known Member

    Messages:
    271
    Likes Received:
    12
    Best Answers:
    0
    Trophy Points:
    138
    #2
    You should have only 1 robots.txt file in root folder.

    You can exclude any folder using that robots.txt file. It doesn't matter whether they are static or dynamic.

    For example, to exclude "sample" folder, use

    Disallow: /sample/
     
    vishwaa, Jul 30, 2007 IP
  3. jingjing

    jingjing Peon

    Messages:
    18
    Likes Received:
    0
    Best Answers:
    0
    Trophy Points:
    0
    #3
    read more in
    robotstxt.org
    robots.txt file should be add in root folder of your site.
     
    jingjing, Jul 30, 2007 IP
  4. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #4
    adding the folder that you want search engine follow to your robots.txt file is better option
     
    trichnosis, Aug 6, 2007 IP