If I have the following robots.txt as an example: User-agent: * Disallow: /search? Code (markup): Would it allow urls like: http://domain.com/search/searchTerm and block the following from being indexed: http://domain.com/search?q=searchTerm
I just did some more searching and found: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449 Apparantly only Google allows wildcards so it's best to use the following: User-agent: Googlebot Disallow: /*search? Code (markup): This will block all url from being indexed that include "*search?"