I've been advised to use the following in my robots.txt file User-agent: * Disallow: /test/ Disallow: ./test/ I believe the 1st disallow means BOTS do not visit anything in the folder TEST but do VISIT the sub folders Does the 2nd one mean BOTS do not visit anything in TEST and any other sub folder. I've read the robots.org page but with subfolders I'm lost. So lets say in a folder called SHOP I have 3 .php files and I want to disallow to one I would do this Disallow: /SHOP/1.php That way 2.php and 3.php would be indexed Also within a webpage does the command NO FOLLOW mean the page will no longer be indexed, but it will follow and links within it? Thanks. Ian
The first one there will disallow that directory and everything in it including sub directories. The second one I'm not sure. I haven't seen or used that method before. That's correct. No, the opposite is true. The page will be indexed, but the links will not be followed.
Disallow cannot contain relative paths (e.g. ./test or test). It may only contain partial absolute paths (e.g. /test will match /test.html and /test/test.html, while /test/ will only match /test/test.html, but not /test.html). Search this forum for robots.txt and you will find numerous discussions and examples about robots.txt. J.D.