Let me explain, i don't want search engines to index forum pages such as: http://forums.digitalpoint.com/forumdisplay.php?f=50 i want them to crawl through forum pages and index all threads such as: http://forums.digitalpoint.com/showthread.php?t=1303772 Is this possible and how can i do this? If i disallow the forum pages, will the search engines continue to index the threads?
This is exactly the question I have. Its simple to disallow any file or directory, such as: disallow: /forums/ disallow: /forums/somefile.php can it allow or disallow urls using some predefined pattern? I mean, I have some urls like this: /forums/index.php?act=Post&CODE=02&f=15&t=13&qpid=15 /forums/index.php?act=Post&CODE=02&f=15&t=13&qpid=85 /forums/index.php?act=Post&CODE=02&f=15&t=14&qpid=16 and I do not want bots to index them, and I want to disallow the urls, which start on: /forums/index.php?act=Post&CODE=02 ? if I write the following in robots.txt disallow: /forums/index.php?act=Post&CODE=02 or disallow: /forums/index.php?act=Post&CODE=02* will it work? or it will disallow the whole /forums/ directory or block /forums/index.php file totally? I do not want to test it till I'm not sure, while google has indexed more than 35000 urls from my webpage and there is a big threat in modifying robots.txt.