Including a folder within a Disallow folder?

Discussion in 'robots.txt' started by Arcos, Jan 15, 2007.

  1. #1
    Does anyone know how to include a folder that is within a Disallowed folder in robots.txt?

    For example

    Disallow: /administration/

    BUT

    I want to allow /administration/pdf

    Thanks for your help.
     
    Arcos, Jan 15, 2007 IP
  2. Jean-Luc

    Jean-Luc Peon

    Messages:
    601
    Likes Received:
    30
    Best Answers:
    0
    Trophy Points:
    0
    #2
    Hi,

    The short answer is that you cannot do that with a standard robots.txt file.

    Some search engines support private extensions, like the Allow: directive which is not standard and which you could use.

    For example:
    User-agent: Googlebot 
    Disallow: /administration/
    Allow: /administration/pdf/
    Code (markup):
    Jean-Luc
     
    Jean-Luc, Jan 16, 2007 IP