Cheat Codes - Ford Cars in Milton Keynes - Acs Student Loans - Online Advertising - Loans

PDA

View Full Version : robots.txt special excusion needed


exam
Dec 11th 2004, 1:58 pm
I have a directory /preferences/ It's ok if the bots spider the index page, but I don't want them to see all the pages where I pass a variable to the index page. All I do is pass the query string in to be able to pass it back out to the seach page, so that the user returns to their original search after setting the prefs. So the all the "pages" that the bot sees (Yahoo is currently spidering hundreds of these pages in the form /preferences/?q=somequery .....

So, I would like to allow /preferences/ but disallow /preferences/?q=xxx

I have had a look at http://www.robotstxt.org/wc/exclusion-admin.html but it doesn't specifically address this issue.

What are your opinions. Would this work?

User-agent: *
Disallow:

User-agent: *
Disallow: /preferences/?

exam
Dec 12th 2004, 3:46 pm
28 views and no replies I guess no one has run into something like this before. At http://www.robotstxt.org/wc/norobots.html around the bottom of the page is says the value of this field [disallow] specifies a partial URL that is not to be visited. This can be a full path, or a partial path; any URL that starts with this value will not be retrieved. For example, Disallow: /help disallows both /help.html and /help/index.html, whereas Disallow: /help/ would disallow /help/index.html but allow /help.html.How is this to be interpreted? Do you think this means my code User-agent: *
Disallow:

User-agent: *
Disallow: /preferences/? will do what I want it too?

The other option would be to change my site code and use a session var for the info.

I'll wait and see if anyone has a comment :)