Does anyone know how to include a folder that is within a Disallowed folder in robots.txt? For example Disallow: /administration/ BUT I want to allow /administration/pdf Thanks for your help.
Hi, The short answer is that you cannot do that with a standard robots.txt file. Some search engines support private extensions, like the Allow: directive which is not standard and which you could use. For example: User-agent: Googlebot Disallow: /administration/ Allow: /administration/pdf/ Code (markup): Jean-Luc