I want to disallow thiese category from indexing. e.g /abc/xyz/2011/ and /abc/zyx/2011/ the red parts are different. Can I use *to represent xyz and zyx in robots?
Some bots will respect it, some won't. Most of the bots respect blocking a particular file type. EG: to block access to all pdf files in your site use... Disallow: /*.pdf$ To block the /2011/ folder in your example, use Disallow: /2011*/ Make sure you test it in Google webmaster tools before uploading it to your site. Not sure how many bots respect that, but most of the major ones should. Further reading http://ghita.org/search-engines-dynamic-content-issues.html http://en.wikipedia.org/wiki/Robots.txt http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449&from=40367&rd=1 Cheers James