Standard Disallowed Pages with Robots.txt

Discussion in 'robots.txt' started by Diablos, Aug 23, 2007.

0
  1. #1
    Hey Guys,

    Just getting into the whole robots.txt control stuff and I was wondering if there is a standard set of pages people usually disallow like /admin for example or /private.
    Obviously the bot can't access these without a password so they should be excluded to prevent errors right?
     
    Diablos, Aug 23, 2007 IP
  2. trichnosis

    trichnosis Prominent Member

    Messages:
    13,785
    Likes Received:
    333
    Best Answers:
    0
    Trophy Points:
    300
    #2
    in my opinion, you must add those pages to your robots.txt
     
    trichnosis, Aug 27, 2007 IP
  3. Directory_SEO

    Directory_SEO Peon

    Messages:
    202
    Likes Received:
    5
    Best Answers:
    0
    Trophy Points:
    0
    #3
    in my blog (sig), i listed some sueful SEO tools, you will see 2 tools for robots.txt file, hopefully you will find it useful
     
    Directory_SEO, Aug 27, 2007 IP