I have a directory I use with some php redirect scripts for cloaking my affiliate urls I don't want indexed by the search engines. I put a robots.txt file in the recomends directory I dont want the bots to index like this. User-agent: * Disallow: /recommends/ Is that the place to put the robots.txt file & would a .htaccess.txt be better & where should ether one be placed at in the root or the directory I don't want index.
Your robots.txt MUST be in your root of the domain... And what do you mean .htaccess? .htaccess has nothing to do with robots.txt
I read some where that not all search engines stop indexing directories with a robots.txt file. If that's the case then what should I put into a .htaccess file to stop the search engines from indexing the directory
Most major search engines listen to robots.txt, smaller search engines that no one really cares about don't...