Hello! I need help in writing a correct robots.txt file. I don't know anything about this and I just want to make sure I end up getting the right codes based on how I want things to be. These are exactly the files and folders inside my server: FILES: - index.html - welcome.html - logo.jpg FOLDERS: - stats - family - sports I have a website inside the "family" and "sports" folder. I also have another one which is the "index.html" file in the root directory. I want robots, agents, spiders, or whatever to be able to crawl everything BUT the website and files I have iside the "family" folder. Those are for my family only and I'd like to keep it private. Everything else, I want it to be seen and to be able to be indexed in search engines. I would really appreciate any help I can get. Thank you! PS: Where do I put the robots.txt file? Do I put it in the root directory?
Hi User-agent: * Disallow: /family/ or similar depending where that folder is stick it in the root as a txt file eg w ww.yourdomain.com/robots.txt and you can check it here http://tool.motoricerca.info/robots-checker.phtml PS try a Google search and you will see loads of help Jamie