I'm just revising my robots.txt file and was thinking should i ban a bot from indexing anything in my folder that I use as menus etc which are called by the php include command? OR would this prevent them from indexing the pages that use those files? Which is every page on my site! Thanks Ian
If you disallow a bot access to a folder with files in it, that folder and those files will no be indexed. If they are included by PHP they will still show up on the pages the bot is allowed to index. You will be fine by disallowing the bot from that folder.