exam
Dec 11th 2004, 1:58 pm
I have a directory /preferences/ It's ok if the bots spider the index page, but I don't want them to see all the pages where I pass a variable to the index page. All I do is pass the query string in to be able to pass it back out to the seach page, so that the user returns to their original search after setting the prefs. So the all the "pages" that the bot sees (Yahoo is currently spidering hundreds of these pages in the form /preferences/?q=somequery .....
So, I would like to allow /preferences/ but disallow /preferences/?q=xxx
I have had a look at http://www.robotstxt.org/wc/exclusion-admin.html but it doesn't specifically address this issue.
What are your opinions. Would this work?
User-agent: *
Disallow:
User-agent: *
Disallow: /preferences/?
So, I would like to allow /preferences/ but disallow /preferences/?q=xxx
I have had a look at http://www.robotstxt.org/wc/exclusion-admin.html but it doesn't specifically address this issue.
What are your opinions. Would this work?
User-agent: *
Disallow:
User-agent: *
Disallow: /preferences/?