This may be a stupid question, but how do I create a robots.txt for my site, www.chinamobilephones.net Does it matter whether I put the file at the root or not? And why would this help the search spiders to find my site?
Just open note pad put in your stuff, save it and name it as "robots.txt" without the quotations of course, upload it to your server and you are done... it is really simple example of the code User-agent: * Disallow: h**p://www.yourdomain.com/contract5.doc Disallow: h**p://www.yourdomain.com/contract6.doc Disallow: /file/example you can show the file path or the url it does not matter... robots.txt talks to search engine algorithm's and tells them not to index certain files... Also do not forget to add this meta tag to all of the pages that you want to be indexed This code gos in the head of your code <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW, ARCHIVE"> <head> right here </head