1. Advertising
    y u no do it?

    Advertising (learn more)

    Advertise virtually anything here, with CPM banner ads, CPM email ads and CPC contextual links. You can target relevant areas of the site and show ads based on geographical location of the user if you wish.

    Starts at just $1 per CPM or $0.10 per CPC.

robots.txt special excusion needed

Discussion in 'robots.txt' started by exam, Dec 11, 2004.

0
  1. #1
    I have a directory /preferences/ It's ok if the bots spider the index page, but I don't want them to see all the pages where I pass a variable to the index page. All I do is pass the query string in to be able to pass it back out to the seach page, so that the user returns to their original search after setting the prefs. So the all the "pages" that the bot sees (Yahoo is currently spidering hundreds of these pages in the form /preferences/?q=somequery .....

    So, I would like to allow /preferences/ but disallow /preferences/?q=xxx

    I have had a look at http://www.robotstxt.org/wc/exclusion-admin.html but it doesn't specifically address this issue.

    What are your opinions. Would this work?

    User-agent: *
Disallow:

User-agent: *
Disallow: /preferences/?
    Code (markup):
     
    exam, Dec 11, 2004 IP
  2. exam

    exam Peon

    Messages:
    2,434
    Likes Received:
    120
    Best Answers:
    0
    Trophy Points:
    0
    #2
    28 views and no replies I guess no one has run into something like this before. At http://www.robotstxt.org/wc/norobots.html around the bottom of the page is says
    How is this to be interpreted? Do you think this means my code 
    User-agent: *
Disallow:

User-agent: *
Disallow: /preferences/?
    Code (markup):
    will do what I want it too?

    The other option would be to change my site code and use a session var for the info.

    I'll wait and see if anyone has a comment :)
     
    exam, Dec 12, 2004 IP