Wildcards and directory depth

Discussion in 'robots.txt' started by bigfishphilip, Jul 25, 2009.

  1. #1
    Hello all,

    I'm using an unusual hierarchy in WordPress to create a structure for a Guide I'm writing. There is a main parent page called My Guide which has child pages for the chapters. Each of those chapters then also have child pages which I'm using to create a directory. So, I end up with URLs like...

    /my-guide/how-does-it-work/this-is-subsection-1
    /my-guide/how-does-it-work/this-is-subsection-2
    /my-guide/why-should-i-use-it/this-is-subsection-1

    I'm hoping to create a single robots.txt directive that allows the top 2 directories only. I could specifically include the chapter names in a bunch of directives, but I'm trying not to have to maintain the robots.txt file along with the guide.

    Is it possible to do something like...

    disallow: /my-guide/*/*

    ...so that these pages still get indexed...

    /my-guide/how-does-it-work/
    /my-guide/why-should-i-use-it/

    Thanks!

    Philip
     
    bigfishphilip, Jul 25, 2009 IP