Hey folks... a few issues if I may... Well, at present my robots.txt shows this only: User-agent: * Disallow: Will this cause me any problems?....I don't think I need anything in particular form it right now, but is it ok to leave this in there?.. and also, I have my blog on my domain but not top level, I have it at http://www.xxxxx.com/myblogishere How should I setup my robots.txt to disallow my WPlogin from being indexed and is there anything else I want to disallow..in your opinion?...and if so, how do I set it up to do so... Thankyou Kindly McrTech
Just enter the relative path of your wp plugin folder at Disallow. You might also want to block your cgi bin using robots.txt. Here is a online Robots.txt generator for you to make things easier. http://www.mcanerin.com/EN/search-engine/robots-txt.asp
User-agent: * Disallow: Right now you are using these instructions in your robots.txt. These instructions mean that each and every crawler can crawl your each and every pages. If you have some private files that you don't want to share over search engines. You can block those in robots.txt like: User-agent: * Disallow: /your private folder or file These instructions will enable your website privacy as no crawler would be able to look into your private folders and files. For url http://www.xxxxx.com/myblogishere, you need to setup a new robots.txt file within the root of the folder. And there you can block the folders and files that you want to disallow like: User-agent: * Disallow: /WPlogin form path In addition you can also block these two folders: Disallow: /cgi-bin Disallow: /search If you need more assistance, feel free to contact me @ my blog